Network traffic management method, system and device based on eBPF and medium

By using eBPF technology for traffic interception and load balancing in the Kubernetes Service network model, combining entropy weight method and slot algorithm, the delay and resource consumption problems in large-scale traffic scenarios are solved, and high-performance traffic governance and load balancing are achieved. Adapt to dynamic environments.

CN118921326AActive Publication Date: 2024-11-08苏州旗开得电子科技有限公司

Patent Information

Application Number
CN202411059327.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-03
Publication Date
2024-11-08
Estimated Expiration
2044-08-03

AI Technical Summary

Technical Problem

The existing Kubernetes Service network model has problems such as high access latency and large resource consumption in large-scale traffic scenarios, which cannot meet the high-performance requirements of cloud platform network traffic governance. The existing load balancing algorithm is complex and does not meet the lightweight requirements. .

Method used

Using eBPF-based network traffic governance method, by intercepting network traffic packets at the kernel mount point, performing traffic interception, load balancing, address conversion and connection tracking, combining the priority load balancing algorithm and slot algorithm with entropy weight method to optimize mapping storage , realize efficient traffic forwarding and load balancing.

Benefits of technology

It realizes high performance and high throughput in cloud platform microservice traffic governance, reduces network latency, avoids load imbalance problems, adapts to dynamic traffic environments, and improves system stability and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921326A_ABST
    Figure CN118921326A_ABST
Patent Text Reader

Abstract

The invention relates to an eBPF-based network flow management method, system and device and a medium, and the method comprises the steps: flow interception: intercepting a network flow data packet at a kernel mounting point from an eBPF program, screening out the network flow data packet accessing a cluster Service, judging whether the destination address of the network flow data packet is a Service address, if yes, sending the network flow data packet to the eBPF program, and if not, sending the network flow data packet to the eBPF program; if the data packet is the network flow data packet of the cluster Service, executing a flow forwarding scheme, and if the data packet is not the network flow data packet, releasing the network flow data packet without influencing other network functions of a host; load balancing: in response to the execution flow forwarding scheme, obtaining Endpoint information of a plurality of real service instances accessing the back end of a cluster Service, and performing Endpoint information selected by a load balancing algorithm; according to the method, the slot algorithm is provided, the number of updating times of Pod meta-information can be effectively reduced, the operation on the storage space is optimized, the system stability is improved, and the method is suitable for application scenes where Endpoints lists are updated massively and frequently.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing technology, and in particular to an eBPF-based network traffic management method, system, device and medium. Background Art

[0002] In recent years, with the rapid development of the Internet economy, cloud computing has become a key link for enterprises to integrate network, computing, storage and other infrastructure, and is an inevitable choice for enterprises to transform digitally. For example, cloud vendors have established large-scale data centers to provide cloud services to a large number of users. More and more companies tend to choose on-demand, elastic and convenient cloud platforms to run their business systems.

[0003] As an important achievement in the development of cloud computing, container virtualization technology implements resource restrictions and view isolation of processes based on Linux's Cgroups and Namespace technologies. Through the sandbox mechanism, processes can run in an independent environment to avoid mutual interference.

[0004] Compared with traditional hypervisor virtualization technology, container virtualization technology is more lightweight, takes up less resources, and is more flexible in achieving application isolation. In 2013, the Docker project packaged the application configuration files and operating environment into images, enabling packaging in one place and running everywhere, solving the problem of portability of cloud platform applications. This innovation led to technological changes in the field of cloud computing infrastructure, and container virtualization technology represented by Docker quickly took a dominant position in cloud platforms.

[0005] With the rise of the Docker project, the concept of container as a service has gradually attracted people's attention. Many companies have launched container orchestration systems to participate in the competition, combining traditional cloud computing infrastructure as a service (IaaS) with platform as a service (PaaS) to provide an operating environment for container applications. Finally, Kubernetes (K8s), a container orchestration platform open sourced by Google in 2014, gradually replaced Swarm, Mesos, Cattle and other projects to become the de facto standard for container orchestration systems with its powerful container orchestration capabilities, support for user-defined components and excellent system architecture.

[0006] The default network traffic forwarding solution of the Service network model of Kubernetes has problems such as high access latency and high resource consumption, and cannot effectively support large-scale traffic application scenarios. Although some existing network solutions have solved the problem of large-scale support, they have ignored the improvement of network performance and portability. eBPF has great potential in network traffic optimization. It breaks through the limitations of user-mode system calls to improve performance, but due to the extremely high security requirements of the operating system kernel, eBPF has many limitations such as limited program size and no support for infinite loops.

[0007] The solutions currently used in the industry can support the basic functions of the Service network model such as traffic forwarding, but the implementation process lacks comprehensive consideration of performance and effectiveness, and there are still problems such as high network latency and high resource usage, which cannot meet the high performance requirements of cloud platform network traffic management. Although the existing load balancing algorithm solves the problem in specific scenarios, the corresponding implementation method and dependent indicators are complex, and do not meet the lightweight requirements of the Service network model traffic management. Summary of the invention

[0008] The purpose of the present invention is to provide a network traffic management method, system, device and medium based on eBPF, which can be seamlessly embedded in the cloud platform, and the traffic originally managed based on iptables rules can be managed by eBPF program, thereby improving the efficiency of traffic forwarding.

[0009] In order to achieve the above-mentioned object, a first aspect of an embodiment of the present disclosure provides a network traffic management method based on eBPF, the method comprising:

[0010] Traffic interception: intercept network traffic data packets from the eBPF program at the kernel mount point, and filter out the network traffic data packets accessing the cluster Service, determine whether the destination address of the network traffic data packet is the Service address, and if it is the network traffic data packet of the cluster Service, execute the traffic forwarding plan, if not, release the network traffic data packet without affecting other network functions of the host;

[0011] Load balancing, in response to the execution of the traffic forwarding solution, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm;

[0012] Address translation, performing address translation on the Endpoint information according to the Endpoint information selected by the load balancing algorithm to complete NAT translation;

[0013] Connection tracking, connection tracking is performed based on the network traffic data packet with address conversion. When the eBPF program receives a response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

[0014] In one embodiment, intercepting the network traffic data packet from the eBPF program at the kernel mount point specifically includes:

[0015] According to different network traffic paths, the cluster service includes the ClusterIP Service network model and the NodePort Service network model;

[0016] Create a socket process, a microservice deployed in a container, and have a corresponding process control group;

[0017] Construct a cgroup network packet filtering mechanism to implement network control for multiple processes in the process control group, intercept connect and send system calls to implement network packet forwarding;

[0018] According to the cgroup network data packet filtering mechanism, the connect and send system calls of the Socket are intercepted at the mount point, the network packet information of the mount point is obtained, and the destination IP and destination Port are extracted;

[0019] According to the ClusterIP Service network model of the cluster, Pod information is obtained in the eBPF map for DNAT. After DNAT processing, the data packet enters the network protocol stack normally and implements network data packet forwarding according to the host routing rules.

[0020] Construct an XDP network packet filtering mechanism; mount the XDP network packet filtering mechanism to the network card, the network driver of the network card, and the kernel network protocol stack, corresponding to three working modes of Offloaded, Native, and Generic from high to low performance;

[0021] The NodePort Service network model of the cluster receives access traffic inside and outside the cloud platform. By opening ports on the host machine and listening to request information, network traffic inside and outside the Kubernetes cluster will reach the node's network card hardware.

[0022] In one embodiment, the NodePort Service network model traffic management of the XDP network packet filtering mechanism includes that the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host and the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on another node;

[0023] The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host as the XDP network packet filtering mechanism, which intercepts the network traffic data packet at the network card receiving layer. If it is a network traffic data packet requesting to establish a connection, a DNAT (destination address translation) process is performed to replace the destination address of the network quintuple with the Pod address selected by the load balancing algorithm, recalculate the header checksum and maintain the conntrack connection tracking information, so that subsequent network traffic is forwarded to the same Pod; if a network traffic data packet requesting to disconnect is received, the conntrack connection tracking information is cleared;

[0024] The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located at another node to intercept the network traffic data packet, execute the DNAT process and perform the SNAT (source address translation) process on the network traffic data packet, and after the SNAT process, query the forwarding information library to update the MAC address of the data packet, and forward the network traffic data packet to the output network card.

[0025] In one implementation, the load balancing algorithm is a priority load balancing algorithm based on the entropy weight method, which uses a kernel-mode user-mode collaborative mechanism to dynamically assess the priority of the current Pod. When the load balancing makes a decision, it relies on kernel-mode information to complete the traffic management process, including:

[0026] Standardize the resource indicators of Pods and divide them into positive indicators and negative indicators.

[0027] The entropy of each indicator is calculated by using the positive and negative indicator calculation formulas to express the impact of the indicators on the score.

[0028] Calculate the redundancy of each indicator, and then calculate the weight of each indicator based on the redundancy, and calculate the priority of each Pod based on the weight.

[0029] A second aspect of the embodiments of the present disclosure provides a network traffic management system based on eBPF, which is applied to the method described, and the system includes:

[0030] The control plane is in user state. The control plane is responsible for subscribing to the Service and Endpoints resource objects of the Kubernetes cluster. After the data processing process, the resource objects after big and small endianness are synchronized to the data plane.

[0031] A data plane, the data plane is located in the kernel state, wherein the data plane includes an eBPF program, an eBPF map, and a kernel mount point, wherein the eBPF program uses the resource object information in the eBPF map as a traffic management basis, and is used to implement a packet forwarding function at the kernel mount point through which the network traffic packet passes;

[0032] The traffic interception module is used to intercept network traffic data packets from the eBPF program at the kernel mount point, and filter out the network traffic data packets accessing the cluster Service, and determine whether the destination address of the network traffic data packet is the Service address. If it is the network traffic data packet of the cluster Service, the traffic forwarding plan is executed, and if not, the network traffic data packet is released;

[0033] Load balancing, in response to the execution of the traffic forwarding solution, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm;

[0034] An address conversion module, used to perform address conversion on the Endpoint information according to the Endpoint information selected by the load balancing algorithm to complete NAT conversion;

[0035] The connection tracking module is used to perform connection tracking based on the network traffic data packet with address conversion. When the eBPF program receives the response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

[0036] In one embodiment, the data plane includes a slot algorithm optimization mapping storage method, the method comprising:

[0037] Split the Endpoints list into single Pod elements for storage;

[0038] In the process of tuning Pod elements, the update granularity of the eBPF map is changed to a single Pod element, and the slot algorithm is used to maintain the one-to-many relationship between Service and Pod;

[0039] During traffic forwarding, a single Pod element is selected as the backend service according to the slot. By marking the elements of Endpoints as deleted, retained, and added, when the elements of the Endpoints list change dynamically, the elements in the three states are processed uniformly.

[0040] In one implementation, the eBPF program is of cgroup type, and the kernel mount point is cgroupv2.

[0041] In one implementation, the NAT conversion includes a DNAT conversion process and a SNAT conversion process.

[0042] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0043] Microcontroller;

[0044] a memory for storing instructions executable by the microcontroller;

[0045] The microcontroller is configured to execute executable instructions stored in the memory to perform the method described in any one of the first aspects.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0047] The present invention provides a network traffic management method, system, device and medium based on eBPF. Compared with the prior art, it has the following beneficial effects:

[0048] 1) The present invention is a method for cloud platform microservice traffic management, which can adapt to the dynamic traffic management environment and meet the requirements of high performance and high throughput.

[0049] 2) The present invention proposes a slot algorithm, which can effectively reduce the number of updates of Pod meta information, optimize the operation of storage space, improve system stability, and is suitable for application scenarios where the Endpoints list is updated in large quantities and frequently.

[0050] 3) The present invention proposes a priority load balancing algorithm based on the entropy weight method. In high-concurrency application scenarios, the priority algorithm implemented based on the entropy weight method can avoid high-load servers and select servers with relatively sufficient resources for load distribution, thereby solving the load imbalance problem under complex requests.

[0051] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0053] Figure 1It is a flowchart of a network traffic management method based on eBPF according to an embodiment of the specification.

[0054] Figure 2 It is a flow chart of the flow management process of the present invention;

[0055] Figure 3 It is a block diagram of a network traffic management system based on eBPF of the present invention;

[0056] Figure 4 It is a flow chart of the present invention for solving ClusterIP type Service traffic management;

[0057] Figure 5 It is a flow chart of solving the NodePort type Service traffic management in the present invention;

[0058] Figure 6 It is a schematic diagram of the slot algorithm implemented by the present invention;

[0059] Figure 7 It is a schematic diagram of the entropy weight method load balancing implemented by the present invention;

[0060] Figure 8 is a network delay comparison bar graph of Embodiment 3 of the present invention;

[0061] Fig. 9 It is a QPS comparison bar chart of the third embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0064] In order to achieve the above-mentioned purpose, the present invention provides a network traffic management method based on eBPF. At present, the default network traffic forwarding solution of the Service network model of Kubernetes has problems such as high access latency and high resource consumption, and cannot effectively support the application scenarios of large-scale traffic. Although some existing network solutions have solved the problem of large-scale support, they have ignored the improvement of network performance and the portability of use. eBPF has great potential in network traffic optimization. It breaks through the limitations of user-mode system calls to improve performance, but due to the extremely high security requirements of the operating system kernel, eBPF has many limitations such as limited program size and no support for infinite loops.

[0065] Embodiment 1:

[0066] Figure 1-Figure 2 It is a flowchart of a network traffic management method based on eBPF according to an embodiment. Using kernel-mode eBPF to replace routing rules requires completing functions such as traffic interception, load balancing, address translation, and connection tracking in sequence. The method includes:

[0067] In step S11, traffic interception is performed, network traffic data packets are intercepted from the eBPF program at the kernel mount point, and the network traffic data packets accessing the cluster Service are screened out, and it is determined whether the destination address of the network traffic data packet is the Service address. If it is the network traffic data packet of the cluster Service, the traffic forwarding solution is executed, and if not, the network traffic data packet is released without affecting other network functions of the host;

[0068] Traffic interception: The eBPF program intercepts network traffic packets at the kernel mount point, filters out network traffic packets accessing the cluster service, distinguishes whether the destination address of the packet is the service address, and executes the traffic forwarding plan if it is the network traffic of the cluster service. If not, the traffic packet is released without affecting other network functions of the host. Write an eBPF program to mount it to the appropriate location of the kernel (such as the tc (Traffic Control) layer or XDP (eXpressData Path) layer of the network stack) to capture and process network traffic packets.

[0069] The steps include:

[0070] Select the mount point:

[0071] XDP: Suitable for very high-performance scenarios, processing data packets directly at the network card driver level, which can reduce CPU usage and latency.

[0072] tc clsact: Slightly higher up the network stack, providing more flexibility and access to packet contents.

[0073] Writing eBPF programs:

[0074] Programs are written using the BPF programming language (based on C).

[0075] The program needs to be able to parse network traffic packets (usually using BPF Helper functions such as bpf_skb_load_bytes to read packet contents).

[0076] Determine whether the destination address of the data packet matches the IP address of the cluster Service.

[0077] Mount the eBPF program:

[0078] Mount the eBPF program to the selected kernel point using the bpf() system call or iproute2 tools such as the tc command.

[0079] Implementation logic:

[0080] If the destination address of the packet is the Service address, load balancing logic is executed (this may require additional eBPF programs or work in conjunction with other kernel modules / user space programs).

[0081] If not, the packet is redirected back to the normal path via bpf_clone_redirect (for XDP) or the corresponding BPF action (such as bpf_redirect for tc clsact), or simply lets the packet continue its normal flow.

[0082] Testing and debugging:

[0083] Use tools such as bpftrace or bpftool to debug and verify the behavior of eBPF programs.

[0084] Ensure network performance is not significantly impacted and verify that traffic is being properly intercepted and redirected.

[0085] In step S12, the load balancing, in response to the execution of the traffic forwarding scheme, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm;

[0086] In step S13, address translation is performed on the Endpoint information selected according to the load balancing algorithm to complete NAT translation;

[0087] Among them, when selecting the Endpoint according to the access policy, address translation should be performed according to the Endpoint to replace the original destination address of the Service network traffic data packet (DNAT). If the backend Endpoint is not the host where the access portal is located, source address translation (SNAT) is also required.

[0088] In step S14, connection tracking is performed based on the network traffic data packet with address conversion. When the eBPF program receives the response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

[0089] Among them, after the address translation, in order to correctly return the response information of the Endpoint to the requesting client, the connection tracking function should be implemented for the data packet that undergoes address translation. When the eBPF program receives the response data packet, it can modify the data packet quintuple according to the connection tracking information and return the expected data packet to the client to prevent the data packet from being dropped.

[0090] Microservices deployed in containers on the cloud platform all have corresponding process control groups. Both parties in network communication need to go through the process of creating a Socket process, initiating a connect system call, establishing a connection, and sending information. ClusterIP Service implements network communication between microservices within the cloud platform. Each step of the communication process is carried out within the cloud platform.

[0091] Therefore, a cgroup network traffic packet filtering mechanism is designed to implement network control for multiple processes in the process control group and intercept system calls such as connect and send to implement network traffic packet forwarding.

[0092] The kernel mount point of the cgroup type eBPF program is cgroupv2. At this time, the network traffic data packet has not yet entered the protocol layer. It has the advantages of no need to maintain connection tracking information and no need to recalculate the checksum of NAT (Network Address Translation). The overhead of implementing traffic management is relatively small.

[0093] In one embodiment, see Figure 3 As shown, the interception of network traffic data packets from the eBPF program at the kernel mount point specifically includes:

[0094] According to different network traffic paths, the cluster service includes the ClusterIP Service network model and the NodePort Service network model; design the traffic forwarding path of the cloud platform Service network model, and design the corresponding optimal mounting points for ClusterIP and NodePort type services respectively;

[0095] Create a socket process, a microservice deployed in a container, and have a corresponding process control group;

[0096] Construct a cgroup network packet filtering mechanism to implement network control for multiple processes in the process control group, intercept connect and send system calls to implement network packet forwarding;

[0097] According to the cgroup network data packet filtering mechanism, the connect and send system calls of the Socket are intercepted at the mount point, the network packet information of the mount point is obtained, and the destination IP and destination Port are extracted;

[0098] According to the ClusterIP Service network model of the cluster, Pod information is obtained in the eBPF map for DNAT. After DNAT processing, the data packet enters the network protocol stack normally and implements network data packet forwarding according to the host routing rules.

[0099] Among them, if it is a ClusterIP Service of the cluster, the Pod (equivalent to the Endpoint element in the Endpoints list) information is obtained in the eBPF map for DNAT (DestinationNetworkAddress Translation). The data packet processed by DNAT enters the network protocol stack normally and is forwarded according to the host routing rules.

[0100] NodePort Service receives access traffic from inside and outside the cloud platform. By opening ports on the host machine and listening to request information, network traffic inside and outside the Kubernetes cluster will reach the node's network card hardware. The network card is the first entry point for receiving network traffic packets, and processing packets at this time has the best performance

[0101] Construct an XDP network packet filtering mechanism; mount the XDP network packet filtering mechanism to the network card, the network driver of the network card, and the kernel network protocol stack, corresponding to three working modes of Offloaded, Native, and Generic from high to low performance;

[0102] The NodePort Service network model of the cluster receives access traffic inside and outside the cloud platform. By opening ports on the host machine and listening to request information, network traffic inside and outside the Kubernetes cluster will reach the node's network card hardware.

[0103] In one embodiment, the NodePort Service network model traffic management of the XDP network packet filtering mechanism includes that the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host and the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on another node; for these two situations, management needs to be performed separately, and the corresponding processes are as follows Figure 4 As shown:

[0104] The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host as the XDP network packet filtering mechanism, which intercepts the network traffic data packet at the network card receiving layer. If it is a network traffic data packet requesting to establish a connection, a DNAT (destination address translation) process is performed to replace the destination address of the network quintuple with the Pod address selected by the load balancing algorithm, recalculate the header checksum and maintain the conntrack connection tracking information, so that subsequent network traffic is forwarded to the same Pod; if a network traffic data packet requesting to disconnect is received, the conntrack connection tracking information is cleared;

[0105] If it is a packet requesting to establish a connection, the DNAT process is executed to replace the destination address of the network quintuple with the Pod address selected by the load balancing algorithm, recalculate the header checksum to prevent the packet from being discarded, maintain the conntrack connection tracking information, and forward subsequent network traffic to the same Pod; if a packet requesting to disconnect is received, the conntrack connection tracking information is cleared;

[0106] The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located at another node to intercept the network traffic data packet, perform a DNAT process on the network traffic data packet and perform a SNAT (source address translation) process, after the SNAT process, query the forwarding information base to update the MAC address of the data packet, and forward the network traffic data packet to the output network card;

[0107] Not only does the DNAT process need to be completed for the data packet, but the SNAT (Source Network Address Translation) process is also required, because the conntrack connection tracking information between different hosts cannot be synchronized. When the host where the Pod is located receives the request and returns the response data packet, it cannot forward it to the requesting client because the connection tracking information corresponding to the destination address of the data packet cannot be found, and the response data packet will be directly discarded. After SNAT, the forwarding information base (FIB) is queried to update the MAC address of the data packet, so that the data packet can be forwarded to the output network card.

[0108] Embodiment 2:

[0109] See also Figure 5 As shown, a network traffic management system based on eBPF is applied to the method described above, and the system includes:

[0110] The control plane is in user state, where the control plane is responsible for subscribing to the Service and Endpoints resource objects of the Kubernetes cluster. After the data processing process, the resource objects after big and small endianness are synchronized to the data plane; the Service traffic path is designed, which is characterized by optimizing the performance of the Service network model through the efficient working mode in the kernel state, and the traffic management function of the routing rules needs to be reconstructed. For the two Service network models of ClusterIP and NodePort, the corresponding network traffic paths are different, and the network traffic forwarding path needs to be designed specifically.

[0111] A data plane, the data plane is located in the kernel state, wherein the data plane includes an eBPF program, an eBPF map, and a kernel mount point, wherein the eBPF program uses the resource object information in the eBPF map as a traffic management basis, and is used to implement a packet forwarding function at the kernel mount point through which the network traffic packet passes;

[0112] The data plane needs to rely on the resource object information synchronized with the control plane when forwarding traffic. The backend of the Service network model is a list of Endpoints consisting of multiple Pods. If the key-value pairs consisting of Service and Endpoints are directly stored in the eBPF map of the data plane, the Pod tuning process will cause the eBPF map to be frequently updated, and the list of Endpoints in the kernel will be constantly created and deleted, resulting in performance issues. Moreover, this logical relationship is not convenient for the load balancing algorithm to select backend services.

[0113] Packet forwarding function, to achieve dynamic load balancing function. The original network model uses a random algorithm to implement the load balancing mechanism. It only relies on the number of connections to evenly distribute traffic to different Pods. This method often leads to load imbalance. Therefore, a new algorithm is proposed to comprehensively consider multiple major factors that cause load changes and ensure relative load balance as much as possible.

[0114] The microservice applications of the cloud platform are located on the nodes of the distributed cluster. The resource status of each node is different. It is more suitable to use a dynamic load balancing algorithm to implement the load balancing mechanism. The time complexity of the load balancing algorithm is directly fed back as the delay time of the request response, so it is not appropriate to use a complex algorithm process. The load balancing process of the Service network model is completed in the kernel state. Due to the limitations of eBPF, too many indicators cannot be synchronized to the kernel. To this end, the CPU utilization rate, memory utilization rate, CPU allocation, and memory allocation of the Pod are used as decision indicators for load balancing, and a priority load balancing algorithm based on the entropy weight method is proposed. The kernel state and user state collaborative mechanism is used to dynamically evaluate the priority of the current Pod. When the load balancing makes a decision, it relies on kernel state information to complete the traffic management process;

[0115] The traffic interception module is used to intercept network traffic data packets from the eBPF program at the kernel mount point, and filter out the network traffic data packets accessing the cluster Service, and determine whether the destination address of the network traffic data packet is the Service address. If it is the network traffic data packet of the cluster Service, the traffic forwarding plan is executed, and if not, the network traffic data packet is released;

[0116] Load balancing, in response to the execution of the traffic forwarding solution, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm;

[0117] An address conversion module, used to perform address conversion on the Endpoint information according to the Endpoint information selected by the load balancing algorithm to complete NAT conversion;

[0118] The connection tracking module is used to perform connection tracking based on the network traffic data packet with address conversion. When the eBPF program receives the response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

[0119] In one embodiment, the data plane includes a slot algorithm optimization mapping storage method, the method comprising:

[0120] Split the Endpoints list into single Pod elements for storage;

[0121] In the process of tuning Pod elements, the update granularity of the eBPF map is changed to a single Pod element, and the slot algorithm is used to maintain the one-to-many relationship between Service and Pod;

[0122] During traffic forwarding, a single Pod element is selected as the backend service according to the slot. By marking the elements of Endpoints as deleted, retained, and added, when the elements of the Endpoints list change dynamically, the elements in the three states are processed uniformly. The idea of ​​minimizing the change of the position of the Pod element is adopted. When the elements of the Endpoints list change dynamically, the elements in the three states are processed uniformly, which solves the problems caused by large granularity and frequent updates when the eBPF mapping data is updated.

[0123] Optimize the mapping storage relationship of the data plane. The data plane needs to rely on the resource object information synchronized by the control plane when forwarding traffic. The backend of the Service network model is a list of Endpoints consisting of multiple Pods. If the key-value pairs consisting of Service and Endpoints are directly stored in the eBPF mapping of the data plane, the Pod tuning process will cause the eBPF mapping to be frequently updated, and the list of Endpoints in the kernel will be constantly created and deleted, resulting in performance issues. Moreover, this logical relationship is not convenient for the load balancing algorithm to select backend services.

[0124] In one implementation, the eBPF program is of cgroup type, and the kernel mount point is cgroupv2.

[0125] In one implementation, the NAT conversion includes a DNAT conversion process and a SNAT conversion process.

[0126] Combine traffic management path design, slot algorithm and dynamic load balancing algorithm to improve the performance of cloud platform microservice traffic management.

[0127] Embodiment three:

[0128] A network traffic management method based on eBPF.

[0129] Implementation of this method involves the following steps:

[0130] 1) Design the traffic forwarding path of the cloud platform Service network model, and design the corresponding optimal mounting points for ClusterIP and NodePort type Services respectively;

[0131] 2) Realize functions such as traffic interception, address translation, load balancing, link tracking, etc.;

[0132] 3) Optimize the efficiency of data information synchronization on the data plane;

[0133] 4) Optimize the server-side load balancing strategy to achieve dynamic load balancing process.

[0134] The overall architecture of the traffic path in step 1 includes a data plane and a control plane. The data plane needs to rely on the resource object information synchronized by the control plane when forwarding traffic. The backend of the Service network model is a list of Endpoints consisting of multiple Pods. If the key-value pairs consisting of Service and Endpoints are directly stored in the eBPF map of the data plane, the tuning process of the Pod will cause the eBPF map to be frequently updated, and the list of Endpoints in the kernel will be constantly created and deleted, resulting in performance problems. Moreover, this logical relationship is not convenient for the load balancing algorithm to select backend services.

[0135] The traffic path design in step 1 optimizes the performance of the Service network model through efficient kernel-mode operation, which requires the reconstruction of the traffic management function of the routing rules. For the two Service network models, ClusterIP and NodePort, the corresponding network traffic paths are different, and the network traffic forwarding paths are designed accordingly.

[0136] The traffic interception described in step 2 is the Service network model object of the cloud platform, which corresponds to the data packet in the kernel. The interception is implemented according to the five-tuple of the data packet (source address, source port, destination address, destination port, protocol); address translation includes source address translation, destination address translation and full translation; link tracking is based on the establishment tracking information of the data packet to ensure that the data packets of the same connection can reach the same destination normally; load balancing ensures that different backend services can evenly receive network traffic and improve network performance.

[0137] The data synchronization described in step 3 corresponds to a slot algorithm to optimize the mapping storage method, split the Endpoints list into single Pod elements for storage, and in the process of continuously tuning the Pod, change the update granularity of the eBPF mapping to a single Pod element, and use the slot to maintain the one-to-many relationship between Service and Pod. When forwarding traffic, a single Pod element is selected as the backend service according to the slot. The slot algorithm marks the elements of Endpoints as deleted, retained, and added, and adopts the idea of ​​minimally changing the position of the Pod element. When the elements of the Endpoints list change dynamically, the elements in the three states are uniformly processed, solving the problem of large granularity and frequent updates when the eBPF mapping data is updated.

[0138] An example of Endpoints list status change is shown in status 1:<A,B,C,D,E,F> , state 2:<A,B,C,D,E,F> To state 3:<G,E,C,D> , where each letter represents a Pod, and the number in the brackets represents the slot of the Pod. When Endpoints is tuned from state 1 to state 2, Pod F is added to Endpoints, and it is assigned a slot number that is the length of the Endpoints list plus "1". When tuning from state 2 to state 3, it includes both deleting Pods and adding Pods. At this time, the slot algorithm gives priority to placing the newly added Pods in the smallest slot of the deleted Pods in turn. If there are still vacancies in the slots where the deleted Pods are located, and the vacant slot number is smaller than the slot number of the retained Pod, the Pod with the largest slot number in the retained Pod will be moved forward to the vacant slot in turn until the vacant slot is filled, or there is no situation where the slot where the retained Pod is located is larger than the vacant slot. The slot algorithm ends. The algorithm principle is as follows Figure 6 As shown,<A,B,F> To delete a slot number,<E,C,D> To reserve the slot number, <g>Indicates the newly added slot number.

[0139] The slot algorithm changes the update granularity of the eBPF map from the Endpoints list to a single Pod element, and reduces the number of updates to the eBPF map. If the slot algorithm is not used, when migrating from state 2 to state 3, it is necessary to delete the old Endpoints containing 6 elements and create new Endpoints containing 4 elements. After the slot algorithm is used, only 2 elements need to be deleted and 2 elements updated, and the algorithm efficiency is greatly improved.

[0140] The microservice applications of the cloud platform are located on the nodes of the distributed cluster. The resource status of each node is different. It is more suitable to use a dynamic load balancing algorithm to implement the load balancing mechanism. The time complexity of the load balancing algorithm is directly fed back to the delay time of the request response, so it is not appropriate to use a complex algorithm process. The load balancing process of the Service network model is completed in the kernel state. Due to the limitations of eBPF, too many indicators cannot be synchronized to the kernel. To this end, this paper uses the CPU usage rate, memory usage rate, CPU allocation, and memory allocation of the Pod as the decision indicators for load balancing, and proposes a priority load balancing algorithm based on the entropy weight method. The kernel state and user state collaborative mechanism is used to dynamically evaluate the priority of the current Pod. When the load balancing makes a decision, it relies on kernel state information to complete the traffic management process.

[0141] The entropy weight method is a mathematical method that measures the impact of an indicator on the comprehensive evaluation based on the degree of data dispersion. The greater the difference between the indicators, the greater the impact on the comprehensive indicator and the greater the weight. The entropy weight method uses the characteristics of entropy to objectively assign values ​​to the discreteness of the data, and achieves a comprehensive score for multiple indicators to distinguish the priorities between samples. Suppose there are n data and m evaluation indicators, where the indicator value is x. ij (i=1,...,n,j=1,..,m) represents. First, the resource indicators of the Pod are standardized and divided into positive indicators and negative indicators. The corresponding formulas are as follows:

[0142]

[0143] Through the calculation formula of positive indicators and reverse indicators, the influence of indicators on the score can be effectively expressed, and then the entropy of each indicator is calculated by the following formula:

[0144]

[0145] Then calculate the redundancy of each indicator, and then calculate the weight of each indicator based on the redundancy, and calculate the priority of each Pod based on the weight. i , and its corresponding formula is as follows:

[0146]

[0147] The entropy weight method uses the CPU and Memory allocation of the Pod as positive indicators, which means that the higher the CPU and Memory allocation, the better; it uses the CPU utilization rate and the Memory utilization rate as negative indicators, which means that the lower the CPU utilization rate and the Memory utilization rate, the better. It can objectively express the relative advantages and disadvantages of the Pod's resource allocation and utilization rate in various situations.

[0148] The process of the priority load balancing algorithm based on entropy weight method is as follows Figure 7 As shown. The control plane obtains the four resource indicators of the cluster Pod from the monitoring module, sorts the indicator data and passes it to the entropy weight priority module. Based on the kernel-mode user-mode collaboration mechanism, the control plane synchronizes the Pod resource object and priority information to the kernel-mode eBPF mapping. When the traffic requesting the Service arrives and is captured by the kernel-mode eBPF program, the traffic is dynamically forwarded to ensure that the Pod load is relatively balanced.

[0149] In view of the eBPF-based Service network model proposed in the above process, this paper constructs corresponding experiments to verify the effectiveness of the network model and the two algorithms. Since the Service network model involves communication between different nodes, it is necessary to build a Kubernetes cluster environment. The experiment is carried out on a 3-node Kubernetes cluster built by the cloud server Entropy. The experimental environment is shown in Table 1 below:

[0150] software Version CPU 4 cores Memory 16G operating system CentOS Kernel 5.10.134-16.1.al8 Kubernetes 1.23.10

[0151] Table 1

[0152] Result analysis:

[0153] In order to test the performance of the eBPF solution when there are large-scale services and pods in the Kubernetes cluster, 10, 20, 50, and 100 ClusterIP and NodePort type services were created in turn, and each service contained 10 backend pods. The eBPF solution and the iptables solution were experimented under the same concurrency pressure.

[0154] Depend on Figure 8-Figure 9 The experimental results show that when the number of services and pods in the cluster gradually increases, the eBPF-based solution grows slowly in terms of network latency, while the iptables solution shows a sharp growth trend; in terms of throughput, the eBPF solution can handle larger QPS. Therefore, the eBPF-based solution implemented in this paper can provide lower network latency and higher throughput when completing traffic management tasks, improving the performance of cloud platform infrastructure for traffic management.

[0155] In step 4, load balancing is that the control plane obtains the four resource indicators of the cluster Pod from the monitoring module, sorts the indicator data and passes it to the entropy weight priority module. The control plane synchronizes the Pod resource object and priority information to the kernel state eBPF mapping based on the kernel state user state collaboration mechanism. When the traffic requesting the Service arrives and is captured by the kernel state eBPF program, dynamic traffic forwarding is performed to ensure that the Pod load is relatively balanced.

[0156] Embodiment 4

[0157] The present disclosure also provides an electronic device, including:

[0158] Microcontroller;

[0159] a memory for storing instructions executable by the microcontroller;

[0160] Wherein, the microcontroller is configured to execute executable instructions stored in the memory to perform any method described in any one of the above-mentioned embodiments.

[0161] The embodiment of the present disclosure further provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method described in any one of the above-mentioned embodiments are implemented.

[0162] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments; within the technical concept of the present disclosure, various changes, modifications, substitutions and variations may be made to these embodiments, and these changes, modifications, substitutions and variations all fall within the protection scope of the present disclosure.

[0163] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction, and they should also be regarded as the contents disclosed in this disclosure. In order to avoid unnecessary repetition, this disclosure will not further describe various possible combinations. The technical scope of this application is not limited to the contents in the specification, and its technical scope must be determined according to the scope of the claims.< / g>

Claims

1. A network traffic management method based on eBPF, It is characterized in that The method comprises: Traffic interception: intercept network traffic packets from the eBPF program at the kernel mount point, and filter out the network traffic packets accessing the cluster Service, determine whether the destination address of the network traffic packet is the Service address, and if it is the network traffic packet of the cluster Service, execute the traffic forwarding plan, otherwise release the network traffic packet; Load balancing, in response to the execution of the traffic forwarding solution, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm; Address translation, performing address translation on the Endpoint information according to the Endpoint information selected by the load balancing algorithm to complete NAT translation; Connection tracking, connection tracking is performed based on the network traffic data packet with address conversion. When the eBPF program receives a response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

2. The method according to claim 1, It is characterized in that The interception of network traffic data packets from the eBPF program at the kernel mount point specifically includes: According to different network traffic paths, the cluster service includes the ClusterIP Service network model and the NodePort Service network model; Create a socket process, a microservice deployed in a container, and a corresponding process control group; Construct a cgroup network packet filtering mechanism to implement network control for multiple processes in the process control group, intercept connect and send system calls to implement network packet forwarding; According to the cgroup network data packet filtering mechanism, the connect and send system calls of the Socket are intercepted at the mount point, the network packet information of the mount point is obtained, and the destination IP and destination Port are extracted; According to the ClusterIP Service network model of the cluster, Pod information is obtained in the eBPF map for DNAT. After DNAT processing, the data packet enters the network protocol stack normally and implements network data packet forwarding according to the host routing rules. Constructing an XDP network packet filtering mechanism; mounting the XDP network packet filtering mechanism to a network card, a network driver of the network card, and a kernel network protocol stack; The NodePort Service network model of the cluster receives access traffic inside and outside the cloud platform, and listens for request information by opening ports on the host.

3. The method according to claim 2, It is characterized in that The NodePortService network model traffic management of the XDP network packet filtering mechanism includes that the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host and the traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on another node; The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located on the same host as the XDP network packet filtering mechanism, which intercepts the network traffic data packet at the network card receiving layer. If it is a network traffic data packet requesting to establish a connection, a DNAT (destination address translation) process is performed to replace the destination address of the network quintuple with the Pod address selected by the load balancing algorithm, recalculate the header checksum and maintain the conntrack connection tracking information, so that subsequent network traffic is forwarded to the same Pod; if a network traffic data packet requesting to disconnect is received, the conntrack connection tracking information is cleared; The traffic accessing the host port passes through the Pod selected by the load balancing algorithm and is located at another node to intercept the network traffic data packet, execute the DNAT process and perform the SNAT (source address translation) process on the network traffic data packet, and after the SNAT process, query the forwarding information library to update the MAC address of the data packet, and forward the network traffic data packet to the output network card.

4. The method according to claim 3, It is characterized in that The load balancing algorithm is a priority load balancing algorithm based on the entropy weight method. It uses the kernel state and user state collaborative mechanism to dynamically evaluate the priority of the current Pod. When the load balancing makes a decision, it relies on the kernel state information to complete the traffic management process, including: Standardize the resource indicators of Pods and divide them into positive indicators and negative indicators. The entropy of each indicator is calculated by using the positive and negative indicator calculation formulas to express the impact of the indicators on the score. Calculate the redundancy of each indicator, and then calculate the weight of each indicator based on the redundancy, and calculate the priority of each Pod based on the weight.

5. A network traffic management system based on eBPF, applied to the method according to any one of claims 1-4, It is characterized in that The system comprises: The control plane is in user state. The control plane is responsible for subscribing to the Service and Endpoints resource objects of the Kubernetes cluster. After the data processing process, the resource objects after big and small endianness are synchronized to the data plane. A data plane, the data plane is located in the kernel state, wherein the data plane includes an eBPF program, an eBPF map, and a kernel mount point, wherein the eBPF program uses the resource object information in the eBPF map as a traffic management basis, and is used to implement a packet forwarding function at the kernel mount point through which the network traffic packet passes; The traffic interception module is used to intercept network traffic data packets from the eBPF program at the kernel mount point, and filter out the network traffic data packets accessing the cluster Service, and determine whether the destination address of the network traffic data packet is the Service address. If it is the network traffic data packet of the cluster Service, the traffic forwarding plan is executed, and if not, the network traffic data packet is released; Load balancing, in response to the execution of the traffic forwarding solution, obtains the Endpoint information of multiple real service instances accessing the cluster Service backend, and performs the Endpoint information selected by the load balancing algorithm; An address conversion module, used to perform address conversion on the Endpoint information according to the Endpoint information selected by the load balancing algorithm to complete NAT conversion; The connection tracking module is used to perform connection tracking based on the network traffic data packet with address conversion. When the eBPF program receives the response to the network traffic data packet, it modifies the data packet quintuple according to the connection tracking information and correctly returns the response information of the Endpoint information to the requesting client.

6. The system according to claim 5, It is characterized in that The data plane includes a slot algorithm optimization mapping storage method, the method including: Split the Endpoints list into single Pod elements for storage; In the process of tuning Pod elements, the update granularity of the eBPF map is changed to a single Pod element, and the slot algorithm is used to maintain the one-to-many relationship between Service and Pod; During traffic forwarding, a single Pod element is selected as the backend service according to the slot. By marking the elements of Endpoints as deleted, retained, and added, when the elements of the Endpoints list change dynamically, the elements in the three states are processed uniformly.

7. The system according to claim 5, It is characterized in that The eBPF program is of cgroup type, and the kernel mount point is cgroupv2.

8. The system according to claim 5, It is characterized in that The NAT conversion includes a DNAT conversion process and a SNAT conversion process.

9. An electronic device, It is characterized in that include: Microcontroller; a memory for storing instructions executable by the microcontroller; The microcontroller is configured to execute executable instructions stored in the memory to perform the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Load balancing method and device based on eBPF architecture and used in K8S

    CN113568711A

  • Multi-cluster load balancing method and device, electronic equipment and storage medium

    CN115987990A

  • Load balancing method, system and device and storage medium

    CN116760834A

  • Redis database cluster proxy system and method based on eBPF

    CN118012569A

  • Discovery of application relationships in clusters

    US11916741B1

Cited By

  • Message transmission method, device and equipment for decoupling control plane and data plane

    CN119697087A

  • EBPF-based key value storage system performance optimization system and method

    CN120929469A

  • Traffic scheduling method, traffic scheduling device, electronic equipment and storage medium

    CN121217667A

  • Traffic scheduling method and device, electronic equipment and storage medium

    CN121217667B