Container cluster management system

Abstract

Embodiments of the present disclosure provide a container cluster management system, comprising multiple working nodes, interception modules, forwarding network bridges, and proxy modules. Each working node comprises a namespace, and the namespace comprises a virtual network interface. The interception modules are disposed at the virtual network interfaces, and are configured to: listen to and intercept data passing through the virtual network interfaces, when the data is non-service data, forward the non-service data to the proxy modules by means of the forwarding network bridges, and when the data is service data, not intercept the service data. The forwarding network bridges are disposed between the namespaces and the working nodes, and are configured to provide communication capabilities between the namespaces and the working nodes, and forward the non-service data intercepted by the interception modules to the proxy modules. The proxy modules are disposed in the working nodes, and are configured to process the received non-service data.

Description

Container cluster management system

[0001] Cross-reference to related applications:

[0002] This disclosure is based on Chinese patent application CN202310770565.7, filed on June 27, 2023, with the invention name “Container Cluster Management System”, and claims the priority of the patent application, and all the disclosed contents are incorporated into this disclosure by reference. Technical Field

[0003] The present disclosure relates to the field of container technology, and in particular to a container cluster management system. Background Art

[0004] There are various management solutions for container clusters, one of which is the Kubenetes system. For example, Kubenetes, also known as K8S, is an open-source system for automating the deployment, scaling, and management of containerized applications. K8S provides custom container networking solutions for various systems via a CNI (Container Network Interface) plug-in.

[0005] Currently, mainstream CNI plug-ins are categorized as routing plug-ins, overlay plug-ins, and underlay plug-ins. However, routing and underlay plug-ins expose the pod (namespace) network segment to external infrastructure (switches / routers). In large-scale cloud platforms with a vast number of pods, this can significantly impact external infrastructure. Overlay plug-ins, due to the additional tunnel encapsulation, can lead to poor container network performance.

[0006] Summary of the Invention

[0007] Based on this, the embodiments of the present disclosure provide a container cluster management system that can process non-business data within the working nodes as much as possible, preventing non-business data from overflowing outside the working nodes, thereby reducing the impact on external infrastructure without reducing the performance of the container network.

[0008] In a first aspect, the present disclosure provides a container cluster management system, the system comprising a plurality of working nodes, the working nodes comprising at least one namespace, the namespace comprising a virtual network interface, the system further comprising: an interception module, a forwarding bridge and an agent module; the interception module being arranged at the virtual network interface, being arranged to monitor and intercept data passing through the virtual network interface, and in the case where the data is non-business data, forwarding the non-business data to the agent module for processing via the forwarding bridge, and in the case where the data is business data, not intercepting the business data; the forwarding bridge being arranged between the namespace and the working node, being arranged to provide communication capability between the namespace and the working node, and forwarding the non-business data intercepted by the interception module to the agent module; the agent module being arranged within the working node, being arranged to process the received non-business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 is a schematic diagram of an Underlay type network of a K8S system in related art;

[0010] FIG2 is a schematic diagram of the structure of an embodiment of a container cluster management system disclosed herein;

[0011] FIG3 is a schematic diagram of an embodiment of a working node of a K8S system in a container cluster management system of the present disclosure;

[0012] FIG4 is a schematic diagram of an embodiment of a K8S system in a container cluster management system of the present disclosure;

[0013] FIG5 is a schematic diagram of an ARP / ND learning process in an embodiment of a K8S system in a container cluster management system of the present disclosure;

[0014] FIG6 is a schematic diagram of updating a high-activity IP address-MAC address in an embodiment of a K8S system in a container cluster management system of the present disclosure;

[0015] FIG7 is a schematic diagram showing the enabling and disabling functions supported by the proxy module in an embodiment of the K8S system in the container cluster management system of the present disclosure.

[0016] Explanation of main components and symbols: 10, working node; 100, namespace; 101, virtual network interface; 102, interception module; 200, forwarding bridge; 300, proxy module. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0018] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0019] In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are used only to facilitate the description of the present disclosure and have no specific meaning. Therefore, "module," "component," or "unit" may be used interchangeably.

[0020] Before introducing the embodiments of the present disclosure in detail, the technical contents related to the present disclosure are first introduced.

[0021] Containers are a technology used to bundle an application and all its necessary files into a runtime environment. As a single unit, containers can be easily moved and run on any operating system in any environment. Containers isolate software, enabling it to run independently on different operating systems, hardware, networks, storage systems, and security policies. This allows container-based applications to seamlessly transition between development, testing, and production environments. Because the operating system isn't packaged into the container, each container requires minimal computing resources, taking up minimal space and being easy to install. By using containers, users can avoid crashes caused by incompatible environments and achieve consistent performance across different machines. Developers can focus on the application itself, rather than having to divert attention to debugging or rewriting it for different server environments. Because containers don't contain an operating system, developers can efficiently deploy them across a cluster, with each container responsible for a single component of a complex application. By placing components in independent containers, developers can also update individual components without rewriting the entire application.

[0022] Applications in containers are isolated from the host machine and do not need to worry about the environment. The biggest difference between containers and virtual machines is that containers do not require an operating system startup cycle, so container startup and shutdown are very fast. Containers can also utilize the host machine's resources very efficiently. Containers package all application languages ​​and frameworks into the container, making them deployable in any environment.

[0023] A container cluster is a collection of cloud resources required to run containers. There are various management solutions for container clusters, with the most popular solutions currently including Swarm, Kubenetes, and Mesos.

[0024] Kubenetes, abbreviated as K8S, is an open source system for automatically deploying, scaling, and managing containerized applications. Because K8S has powerful functions and a set of overall container solutions, it is easier to use. K8S is more active than Swarm, Mesos, etc.

[0025] The following uses Kubernetes as an example to illustrate the technical issues encountered in related technologies. Kubernetes is a mainstream container orchestration system in the Cloud Native Computing Foundation (CNCF). Mainstream cloud vendors and operators are researching and using Kubernetes. The smallest granularity for microservice management in Kubernetes is the pod, with each pod representing an independent network namespace. The Kubernetes network model is implemented by the container runtime on each worker node, using the CNI (Container Network Interface) plugin to manage its network and security functions.

[0026] Kubernetes provides customized container networking solutions for various systems through CNI plugins. CNI plugins can be implemented based on the needs of different scenarios and vendors, and many CNI plugins have been open-sourced on GitHub. Currently, the mainstream CNI plugins are divided into the following categories:

[0027] Routing plugins: Pods on different worker nodes use different address segments. By configuring detailed routes to the target Pods on the worker node, cross-worker node Pod communication is achieved through routing forwarding. A typical routing CNI plugin is calico.

[0028] Overlay plug-ins overlay additional tunnels on the worker node network. This allows pods to send data across worker nodes without affecting the external infrastructure. Overlay container network plug-ins are well decoupled from the infrastructure. A typical overlay CNI plug-in is the flannel plug-in.

[0029] Underlay plug-ins enable pods to communicate directly with external devices via Layer 2 (L2) networking. Specifically, pods can broadcast their own addresses via ARP / ND (Address Resolution Protocol / Neighbor Discovery Protocol) and learn the addresses of other worker pods, enabling point-to-point communication based on the learned IP addresses. Typical underlay CNI plug-ins include macvlan plug-ins and ipvlan plug-ins.

[0030] ARP stands for "Address Resolution Protocol." ARP is a universal address resolution protocol that provides a dynamic mapping between network layer addresses (IP addresses) and physical addresses (MAC addresses). Its purpose is to convert known IP addresses into MAC addresses during data transmission. Why is it necessary to obtain a MAC address from an IP address? When a network device wants to send data to another device, it must know the other device's network layer address (i.e., IP address). However, the IP address alone is not enough. IP datagrams must be encapsulated into frames to be sent over the data link. Data frames must contain the destination MAC address, so the sender must also obtain the destination MAC address. This process of obtaining the destination MAC address from the destination IP address is accomplished by the ARP protocol. The ARP protocol works as follows: Before sending a message, a host first checks its ARP cache to see if there is a corresponding destination host. If not, the host broadcasts an ARP request containing the destination IP address to all hosts on the local network and receives a response to determine the destination MAC address. After receiving the response, the host stores the IP and MAC addresses in the local ARP cache for a specified period of time. The next request directly queries the ARP cache to conserve resources. In IPv4 (Internet Protocol version 4), the ARP protocol can be used to complete the conversion between IP addresses and MAC addresses; in IPv6 (Internet Protocol version 6), the NDP protocol can be used to complete the conversion between IP addresses and MAC addresses.

[0031] As shown in Figure 1, the Kubernetes system includes worker nodes and switches. Each worker node includes a pod (PoD) and a bridge for L2 networking. All pods in a worker node communicate with pods in other worker nodes through the bridge, the worker node, and the switch. Worker node 1 includes one pod with an IP address of 192.168.1.13 / 24, worker node 2 includes two pods with IP addresses of 192.168.1.12 / 24 and 192.168.1.11 / 24, and worker node 3 includes one pod with an IP address of 192.168.1.10 / 24. Worker nodes 1, 2, and 3 all belong to local area network VLAN 100.

[0032] Figure 1 illustrates the use of underlay networking. PODs on the same network are all in the same broadcast domain. When PODs communicate with each other, if the local MAC address of the destination IP address is not available, they must learn the other party's MAC address through ARP / ND. The switch also parses the ARP / ND data and updates the relationship between the MAC address and the switch port. With existing underlay solutions, all POD broadcast messages are visible to external infrastructure (such as switches). This can impact the performance and support capabilities of external infrastructure when the microservices POD scale is large.

[0033] Routing plugins may impact infrastructure routing configurations because the pod network segments are exposed to external infrastructure (switches / routers). Underlay plugin pod IP addresses and ARP / ND broadcasts are also exposed to external infrastructure. On large-scale cloud platforms, where Kubernetes manages tens of thousands of worker nodes and hundreds of thousands or even millions of application pods, routing and underlay plugins can significantly impact external infrastructure. The relatively decoupled overlay plugin, however, suffers from poor container network performance due to the additional tunnel encapsulation.

[0034] The disclosed embodiment provides a container cluster management system. Since an agent module is provided in the working node and an interception module is provided at the virtual network interface of the namespace of the working node, the interception module can intercept non-business data passing through the virtual network interface and forward the intercepted non-business data to the agent module in the working node for processing. Therefore, the non-business data in the working node can be retained in the working node for processing as much as possible, and the non-business data in the working node can be avoided from overflowing to outside the working node for processing, thereby reducing the workload of the external infrastructure and reducing the impact on the external infrastructure. Especially when the cloud platform is extremely large in scale and the number of PODs is extremely large, the impact on the external infrastructure can be greatly reduced without reducing the performance of the container network.

[0035] The following is a detailed description of the embodiments of the present disclosure.

[0036] 2 is a schematic structural diagram of an embodiment of a container cluster management system disclosed herein. The system includes multiple working nodes 10 . The working nodes 10 include at least one namespace 100 . The namespace 100 includes a virtual network interface 101 .

[0037] The systems of the embodiments of the present disclosure include, but are not limited to, Swarm, Kubenetes, and Mesos. A worker node 10 is a machine in the system, which can be a virtual machine or a physical machine. A worker node 10 can run containers, be responsible for executing the actual work of the application, or run a system agent to monitor and manage the namespace on the worker node and the programs running in the namespace, and report the status of the worker node.

[0038] Namespace 100 is a way to organize a container cluster into virtual subclusters. It provides an isolation mechanism that can isolate different processes or threads in different environments, thereby achieving isolation and communication between different environments. Virtual network interface 101 is a virtualization technology. Each virtual network interface can have an independent IP address and can communicate with other virtual network interfaces or physical network interfaces. Virtual network interfaces 101 can be isolated through namespaces, thereby achieving isolation and communication between different virtual networks.

[0039] The working node 10 can create one namespace 100 or more than one namespace 100. The command space 100 can create one container or more than one container.

[0040] The system further includes: an interception module 102 , a forwarding bridge 200 , and a proxy module 300 .

[0041] The interception module 102 is disposed at the virtual network interface 101 of the command space 100 of the working node 10 and is configured to monitor and intercept data passing through the virtual network interface 101, wherein the data passing through the virtual network interface 101 includes both data forwarded from the local namespace 100 to other namespaces 100 through the virtual network interface 101 and data forwarded from other namespaces 100 to the local namespace 100 through the virtual network interface 101. If the data is non-business data, the non-business data is forwarded to the proxy module 300 of the working node 10 for processing via the forwarding bridge 200 of the working node 10. If the data is business data, the business data is not intercepted; not intercepting the business data may mean forwarding the business data to the local namespace 100 via the virtual network interface 101 or forwarding the business data to other namespaces 100 via the virtual network interface 101. Non-business data may refer to data not related to business, including but not limited to: broadcast data, request data, response data corresponding to request data, announcement data, control data, etc. Business data may refer to data related to business.

[0042] The interception module 102 includes but is not limited to: extended Berkeley packet filter eBPF, windows packet capture WinPcap (windows packet capture), SystemTap, DTrace, etc.

[0043] The forwarding bridge 200 is arranged between the namespace 100 of the working node 10 and the working node 10, and is configured to provide communication capabilities between the namespace 100 and the working node 10, that is, the namespace 100 within the working node 10 can communicate with the working node 10 through the forwarding bridge 200. In this way, the namespace 100 of the working node 10 can communicate with the namespace outside the working node 10 through the forwarding bridge 200 within the working node 10. When the working node 10 has more than two namespaces 100, since the more than two namespaces 100 can communicate with the forwarding bridge 200, the more than two namespaces 100 within the working node 10 can communicate with each other through the forwarding bridge 200. In this way, normal communication requirements between the namespaces 100 can be guaranteed. The forwarding bridge 200 also forwards the non-business data intercepted by the interception module 102 to the proxy module 300.

[0044] The proxy module 300 is disposed in the working node 10 and is configured to process the received non-business data. That is, the non-business data related to the namespace 100 in the working node 10 is forwarded to the proxy module 300 of the working node 10 for processing, thereby preventing the non-business data from overflowing outside the working node 10.

[0045] Compared with the related art in which non-business data in the namespace of the working node is overflowed to the outside of the working node for processing, the embodiment of the present disclosure is provided with an agent module in the working node and an interception module at the virtual network interface of the namespace of the working node. The interception module can intercept the non-business data passing through the virtual network interface and forward the intercepted non-business data to the agent module in the working node for processing. Therefore, the non-business data in the working node can be retained in the working node for processing as much as possible, avoiding the non-business data in the working node from overflowing to the outside of the working node for processing, thereby reducing the workload of the external infrastructure and reducing the impact on the external infrastructure. Especially when the cloud platform is extremely large and the number of PODs is extremely large, the impact on the external infrastructure can be greatly reduced without reducing the performance of the container network.

[0046] In some embodiments, the system includes a Kubernetes (Kubernetes) system, and the command space includes a Pod. Kubernetes offers powerful functionality, a comprehensive container solution, and ease of use. Kubernetes is more active than other systems. A Pod is the smallest control unit in the Kubernetes system. Containers run within a Pod, which contains at least one container. Kubernetes controls containers by controlling the Pod. The command space isolates the Pod's operating environment.

[0047] In some embodiments, the interception module 102 includes an extended Berkeley Packet Filter (eBPF).

[0048] eBPF is a very flexible and powerful kernel technology that can be used in a variety of application scenarios. The common uses of eBPF are very wide, and it can be used in a variety of application scenarios such as network monitoring, security filtering, performance analysis and virtualization. The working principle of eBPF is mainly divided into three steps: loading, compiling and executing. eBPF needs to run in the kernel: this is usually done by a user-mode application, which loads the eBPF program through a system call. During the loading process, the kernel copies the code of the eBPF program to the kernel space. The eBPF program needs to be compiled and executed: this is usually done by the Clang / LLVM compiler. After the bytecode is formed, the user-mode bytecode is loaded into the kernel, and a JIT compilation step is used to convert the program's general bytecode into a machine-specific instruction set to optimize the program's execution speed.

[0049] In some embodiments, the interception module 102 is further configured to forward the first non-business data to the proxy module for processing when the non-business data is first non-business data, and not intercept the second non-business data when the non-business data is second non-business data.

[0050] In actual applications, some non-business data may not require additional processing, and some non-business data, if intercepted, may affect the business data interaction between namespaces. Therefore, in order to avoid the problems of additional resource consumption and impact on business data interaction caused by intercepting all non-business data, in the embodiment of the present disclosure, non-business data is divided into first non-business data that needs to be intercepted and forwarded to the proxy module for processing, and second non-business data that does not need to be intercepted and forwarded to the proxy module for processing. That is, when the non-business data is the first non-business data, the interception module forwards the first non-business data to the proxy module for processing, and when the non-business data is the second non-business data, the interception module does not intercept the second non-business data.

[0051] In some embodiments, the first non-business data includes first ARP / ND request data, and the second non-business data includes first ARP / ND response data. The first ARP / ND request data is used by the sending end to inquire about the destination MAC address corresponding to the destination IP address carried in the first ARP / ND request data, and therefore generally requires the receiving end to respond accordingly. The sending end of the first ARP / ND response data generally first receives the corresponding request data. The first ARP / ND response data is the sending end's response data to the corresponding request data, responding to the destination MAC address corresponding to the destination IP address. Therefore, the first ARP / ND response data generally does not require the receiving end to respond. If the first ARP / ND response data is intercepted and forwarded to the proxy module, additional resources will be consumed. Therefore, in the present embodiment, the first ARP / ND request data is treated as non-business data that needs to be intercepted and forwarded to the proxy module, and the first ARP / ND response data is treated as non-business data that does not need to be intercepted or forwarded to the proxy module.

[0052] At this time, the proxy module 300 is further configured to process the first ARP / ND request data according to a first correspondence between the first IP address in the first ARP / ND request data and the locally cached IP address-MAC address.

[0053] A namespace can communicate with other namespaces within the current working node, with other namespaces outside the current working node within the system, and with namespaces outside the system. Therefore, the first correspondence between IP addresses and MAC addresses cached locally by the proxy module may include both correspondences between IP addresses and MAC addresses within the system and correspondences between IP addresses and MAC addresses outside the system.

[0054] The proxy module caches the first correspondence between the IP address and the MAC address locally, which can avoid frequent and repeated learning of the correspondence between the IP address and the MAC address that has been learned before, and helps to improve processing efficiency.

[0055] In some embodiments, the proxy module processes the first ARP / ND request data according to the first correspondence between the first IP address in the first ARP / ND request data and the locally cached IP address-MAC address. There are two different processing methods, which are described in detail below:

[0056] (1) The proxy module is configured to generate corresponding second ARP / ND response data when the first MAC address corresponding to the first IP address is queried from the first correspondence, and send the second ARP / ND response data to the sender of the first ARP / ND request data.

[0057] The first IP address is the destination IP address carried in the first ARP / ND request data. The first ARP / ND request data is used to inquire about the destination MAC address corresponding to the first IP address. If the first MAC address corresponding to the first IP address (i.e., the destination MAC address) is queried in the first corresponding relationship of the local cache, the second ARP / ND response data can be sent directly to the sender of the first ARP / ND request data. The second ARP / ND response data will carry the first MAC address corresponding to the first IP address.

[0058] (2) The proxy module is configured to generate and broadcast corresponding second ARP / ND request data when the first MAC address corresponding to the first IP address is not queried from the first correspondence, update the first correspondence based on the third ARP / ND response data corresponding to the received second ARP / ND request data, generate fourth ARP / ND response data corresponding to the first ARP / ND request data, and send the fourth ARP / ND response data to the sender of the first ARP / ND request data.

[0059] If the first MAC address (i.e., the destination MAC address) corresponding to the first IP address is not found in the first correspondence relationship of the local cache, in order to be able to respond to the first ARP / ND request data, the proxy module needs to first learn the first MAC address corresponding to the first IP address, and thus generates the corresponding second ARP / ND request data and broadcasts the second ARP / ND request data. The second ARP / ND request data carries the first IP address and is used to inquire about the destination MAC address corresponding to the first IP address. If the proxy module receives the third ARP / ND response data corresponding to the second ARP / ND request data, it updates the first correspondence relationship according to the third ARP / ND response data, i.e., adds the first MAC address corresponding to the first IP address to the first correspondence relationship to avoid subsequent re-learning; at the same time, it generates the fourth ARP / ND response data corresponding to the first ARP / ND request data and sends the fourth ARP / ND response data to the sender of the first ARP / ND request data. The fourth ARP / ND response data will carry the first MAC address corresponding to the first IP address.

[0060] In some embodiments, the first non-business data further includes ARP / ND announcement data. The ARP / ND announcement data is non-business data in which the sender of the ARP / ND announcement data broadcasts its own second IP address and second MAC address, and is typically announced by broadcast after the sender is assigned the second IP address and second MAC address.

[0061] At this time, the proxy module is further configured to process the ARP / ND announcement data according to a second correspondence between the second IP address, the second MAC address in the ARP / ND announcement data and the locally cached IP address-MAC address in the system.

[0062] The proxy module locally caches the second correspondence between the IP address and the MAC address within the system. On the one hand, it can avoid frequent and repeated learning, which helps to improve processing efficiency. On the other hand, it can also be compared with the IP address outside the system to determine whether the IP address within the system conflicts with the IP address outside the system.

[0063] In some embodiments, the proxy module processes the ARP / ND announcement data according to the second IP address, the second MAC address in the ARP / ND announcement data, and the second correspondence between the IP address and the MAC address in the local cache in the system. The processing may be performed in the following three different manners, as described in detail below:

[0064] (1) The proxy module is configured to discard the ARP / ND announcement data when it is determined, based on the second correspondence, that the second IP address and the second MAC address are IP addresses and MAC addresses within the system.

[0065] The second IP address and the second MAC address are the IP address and MAC address carried in the ARP / ND announcement data. If the second IP address and the second MAC address are determined to be the IP address and MAC address within the system based on the second correspondence within the system, it can be said that the second IP address and the second MAC address themselves are within the second correspondence within the system, and therefore the ARP / ND announcement data is discarded.

[0066] (2) The proxy module is configured to update the locally cached third correspondence between the IP address and the MAC address outside the system based on the second IP address and the second MAC address when it is determined based on the second correspondence that the second IP address and the second MAC address are not the IP address and the MAC address within the system, and the second IP address does not conflict with the IP address within the system.

[0067] If it is determined based on the second correspondence within the system that the second IP address and the second MAC address are not the IP address and MAC address within the system, it can be said that the second IP address and the second MAC address are IP addresses and MAC addresses outside the system, and the second IP address does not conflict with the IP address within the system, which means that the second IP address outside the system is different from the IP address within the system. Since ARP / ND announcement data is usually the data of the sender of the ARP / ND announcement data being assigned a second IP address and a second MAC address and then broadcasting its own second IP address and second MAC address, the third correspondence between the IP address and the MAC address outside the system locally cached by the proxy module does not cache the second IP address and the second MAC address, so it is necessary to update the third correspondence, that is, to add the second IP address and the second MAC address to the third correspondence to avoid frequent and repeated learning in the future and improve processing efficiency.

[0068] (3) The proxy module is configured to generate response data corresponding to the ARP / ND announcement data when it is determined, based on the second corresponding relationship, that the second IP address and the second MAC address are not the IP address and MAC address within the system, and that the second IP address conflicts with the IP address within the system, and send the response data corresponding to the ARP / ND announcement data to the sender of the ARP / ND announcement data to notify that the second IP address conflicts with the IP address in the namespace within the system.

[0069] If it is determined based on the second correspondence within the system that the second IP address and the second MAC address are not the IP address and MAC address within the system, it can be said that the second IP address and the second MAC address are the IP address and MAC address outside the system, and the second IP address conflicts with the IP address within the system, which means that the second IP address outside the system is the same as the IP address within the system. If a sender subsequently inquires about the destination MAC address corresponding to the same IP address, a different and conflicting destination MAC address may be given, which may cause a communication conflict. Therefore, response data corresponding to the ARP / ND announcement data is generated, and the response data corresponding to the ARP / ND announcement data is sent to the sender of the ARP / ND announcement data to notify the sender of the ARP / ND announcement data that the second IP address conflicts with the IP address in the namespace within the system.

[0070] In some embodiments, the system further comprises a node management component, wherein the node management component is disposed in the working node. The node management component may be configured to manage the working node.

[0071] The interception module is further configured to count the target IP addresses in the data whose activity of sending and receiving is greater than a preset activity, and send the target IP addresses to the node management component; the node management component is configured to send the received target IP addresses to the proxy module; the proxy module is further configured to determine a fourth correspondence between the target IP address and the target MAC address based on the first correspondence between the target IP address and the locally cached IP address-MAC address, and return the fourth correspondence to the node management component; the node management component is further configured to send the received fourth correspondence to the namespace within the working node.

[0072] In the embodiment of the present disclosure, the target IP address whose sending and receiving activity in the statistical data of the interception module is greater than the preset activity, the node management component (for example: the CNI plug-in in the K8S system) collects the target IP address whose activity in the statistics of the interception module is greater than the preset activity, and updates the collected target IP addresses to the proxy module, the proxy module sorts out the fourth correspondence between the active target IP address and the target MAC address, and sends the fourth correspondence to the node management component, and the node management component sends the active fourth correspondence to the namespace (for example, POD) in the working node. In this way, for some high-frequency access target IP addresses, the namespace (for example, POD) no longer needs to perform ARP / ND learning and directly sends and receives IP packets, which can greatly reduce the first packet data delay and also reduce the broadcast volume in the system.

[0073] In some embodiments, the proxy module is further configured to monitor the fourth correspondence, and when a change in the fourth correspondence is detected, update the fourth correspondence and return the updated fourth correspondence to the node management component; the node management component is further configured to send the received updated fourth correspondence to the namespace within the working node. In this way, the namespace (e.g., POD) can always maintain the latest and active fourth correspondence.

[0074] In some embodiments, the node management component is also configured to monitor the status of the proxy module, and when it is monitored that the status of the proxy module is abnormal, control the interception module to be closed, control the namespace within the working node to clear the fourth corresponding relationship, and when it is monitored that the proxy module with an abnormal status returns to normal, control the interception module to be turned on.

[0075] In the embodiment of the present disclosure, the node management component monitors the status of the proxy module, and the monitoring method includes observing the running status of the node management component, and attempting to send an ARP / ND request to the node management to obtain the IP address-MAC address relationship, etc. If the node management component does not run normally or it cannot respond to the ARP / ND request, it is determined that the status of the proxy module is abnormal. After monitoring the abnormal status of the proxy module, the disabling function is implemented, that is, the interception function of the interception module in the namespace of the current working node where the proxy module is located is turned off, and the namespace of the working node where the proxy module with the abnormal status is located processes and learns ARP / ND related data by itself, and at the same time clears the fourth corresponding relationship (such as the ARP table) of the local cache of all namespaces of the current working node. After monitoring the abnormal status of the proxy module to return to normal, the enabling function is implemented, that is, the interception function of the interception module in the namespace of the current working node where the proxy module is located is turned on.

[0076] In some embodiments, the proxy module is further configured to monitor the aging time of the IP address-MAC address in the third correspondence between the IP address and the MAC address outside the system that is locally cached. When it is monitored that the aging time exceeds the preset time and the IP address-MAC address whose aging time exceeds the preset time is not in the fourth correspondence, the IP address-MAC address whose aging time exceeds the preset time is cleared from the third correspondence to obtain an updated third correspondence.

[0077] Compared with the communication within the system, the frequency or activity of the communication between this system and the outside of the system will be much less. If the third correspondence between the IP address and the MAC address outside the system only increases and does not decrease, it will occupy storage space and reduce the search efficiency. Therefore, for the third correspondence between the IP address and the MAC address outside the system, the proxy module can record the creation time when creating each third correspondence between the IP address and the MAC address outside the system, and then monitor the aging time of the IP address-MAC address in the third correspondence. If the aging time exceeds the preset time and the IP address-MAC address whose aging time exceeds the preset time is not in the active fourth correspondence, the IP address-MAC address whose aging time exceeds the preset time is cleared in the third correspondence to obtain an updated third correspondence.

[0078] In some embodiments, the forwarding bridge is further configured to forward the non-business data through the network interface of the working node when the MAC address of the namespace of the working node is the same as the MAC address of the network interface of the working node.

[0079] In this way, the non-service data of the working node can be learned by the external L2 switch, so that the switch can update the mapping relationship between the corresponding MAC and the switch port (ie, the MAC-port table).

[0080] The system of the embodiment of the present disclosure is described in detail below using the K8S system as an example.

[0081] The technical problem to be solved by the embodiments of the present disclosure in the K8S container network is: using the traditional Underlay type network solution, although the network communication performance is good, a large number of ARP / ND broadcasts will cause switch table overflow and cause broadcast storms.

[0082] As shown in Figure 3, in the working node of the K8S system, the main technical means used in the embodiment of the present disclosure are:

[0083] (1) Forwarding bridge: provides communication capabilities between the POD virtual network interface and the network interface of the working node, and provides L2 networking capabilities between PODs. When the MAC address used by the POD is the same as the MAC address of the network interface of the working node, it can select the target POD according to the IP address of the data packet and forward the data packet to the virtual network interface of the corresponding target POD. It also forwards the data packet intercepted by eBPF to the proxy module.

[0084] (2) eBPF: The eBPF program is loaded on the POD virtual network interface. The eBPF program intercepts ARP / ND related messages and performs the following processing actions: if it is an ARP / ND request packet or ARP / ND announcement, the ARP request packet or ARP / ND announcement is forwarded to the proxy module of this working node; if it is an ARP / ND response packet, no interception is performed.

[0085] (3) Proxy module: Each K8S minion worker node hosts an independent proxy module. The features of this proxy module include:

[0086] (a) Maintaining the relationship between IP address and MAC address:

[0087] The relationship between the IP address and MAC address of the POD and the working node in the system is locally saved and updated. If a POD in the system is reborn, newly created, or deleted, the agent module monitors the life cycle of the POD in real time and updates the relevant information.

[0088] For the relationship between IP addresses and MAC addresses outside the system, the local configuration is updated after learning ARP / ND packets.

[0089] Supports the aging mechanism and regular update function of the IP address-MAC address relationship table outside the system. When the IP address-MAC address table entry is not used for a certain period of time, the corresponding record will be deleted.

[0090] (b) Proxy ARP / ND message processing:

[0091] When an ARP / ND request message is received, the MAC address information corresponding to the IP address in the request message is queried from the local cache. If the corresponding MAC address information exists, an ARP / ND response message is encapsulated and responded to the corresponding requester. If the corresponding MAC address information does not exist, an ARP / ND request message broadcast learning is initiated. After receiving the corresponding response, the local IP address-MAC address relationship is updated, and an ARP / ND response is constructed and returned to the requester.

[0092] When an ARP / ND announcement message is received and the IP address and MAC address in the announcement message are found from the local cache to be outside the system, if the IP address in the announcement message does not conflict with the IP address in this system, the local IP-MAC relationship is updated. If the IP address in the announcement message conflicts with the IP address in this system, a response message is constructed to inform the announcer of the IP address conflict.

[0093] (c) Supports the enabling and disabling functions of the proxy module:

[0094] (d) It has intelligent analysis function, which can count the IP addresses and MAC addresses that are currently used most frequently, so that POD can update the IP addresses and MAC addresses that are used most frequently.

[0095] (e) The proxy module provides MAC address filtering policy functions to control the ARP / ND learning of these MACs.

[0096] The embodiment of the present disclosure utilizes eBPF to intercept broadcast packets for POD and forward them to the proxy module in the working node for processing and local response. Through the system of the embodiment of the present disclosure: (1) the broadcast packets caused by a large number of PODs in the working node are terminated in the working node, which will not cause capacity and performance impact on external infrastructure equipment (such as switches); (2) the ARP / ND packets of POD are automatically intercepted by eBPF, which has no impact on the application; (3) the ARP / ND request packets are responded locally, which will greatly speed up the response time of ARP / ND response packets for cross-nodes and improve the user's first packet processing performance; (4) the broadcast suppression function based on the characteristic MAC can be customized; (5) using Underlay technology, POD can interact with external network equipment L2, with high forwarding performance, and in the hybrid cloud scenario of containers and virtual machines, it can support point-to-point direct communication between containers and virtual machines.

[0097] The application environment of the embodiment of the present disclosure includes: (1) a container cluster based on K8S, and (2) the working nodes in the cluster use the Linux system, and the kernel version supports the eBPF function.

[0098] As shown in Figure 4, the system of the embodiment of the present disclosure includes the following functional modules: proxy module, eBPF, forwarding bridge, and CNI plug-in (i.e., node management component); among them, the proxy module, eBPF, and forwarding bridge are data plane modules, and CNI is a configuration plane module. The details are as follows:

[0099] CNI plug-in: manages and configures the container network. It is called by the K8S core component kubelet (in the K8S system, kubelet is the main "node agent" running on each working node and is a core component of the K8S framework) when the POD is created. It configures the IP / MAC / routing information of the container network port for the POD microservice, and configures the container network port to the corresponding forwarding bridge. It also supports loading and unloading eBPF modules.

[0100] eBPF: A POD-level program loaded on the POD network port to intercept, analyze, and process data packets passing through the network port.

[0101] Forwarding bridge: Multiple POD network ports of a working node can be aggregated to one network port of the working node through a forwarding bridge. The forwarding bridge provides forwarding support when the MAC address of the POD is the same as the MAC address of the working node's network port. This is mainly because, in L2 networking, the cross-node L2 switch needs to learn which network port the MAC address of the forwarded data is forwarded from. This learning process is learned by the switch through ARP / ND. If ARP / ND suppression is used within the working node to limit ARP / ND from the working node, it may cause the switch to be unable to learn the correspondence between its forwarding MAC and the switch port. Therefore, the forwarding bridge must support forwarding to the corresponding POD based on the IP address when the MAC address of the POD is the same as the MAC address of the working node's network port. The advantage of this is that the ARP / ND message of the working node's network port IP will be learned by the L2 switch, and the mapping relationship between the corresponding MAC and switch port (MAC-port table) will be updated.

[0102] Proxy module, a node-level component, each working node is equipped with a proxy module, which is mainly used to centrally process the broadcast packets of the current working node.

[0103] Scenario 1, normal ARP / ND learning process, as shown in Figure 5:

[0104] 1. When POD sends data to an external destination IP address, it needs to learn the destination MAC address corresponding to the destination IP first. If there is no corresponding record in the local ARP table of the kernel, it will send an ARP / ND request packet to broadcast the destination MAC address corresponding to the learned destination IP address.

[0105] 2. The eBPF program on the POD network port intercepts the data sent and received by the POD (step S1). For non-RP / ND related data, no additional processing or interception is performed (step S2). For ARP / ND related data, if the ARP / ND related data is ARP / ND request data, the data is forwarded to the proxy module in the working node (step S3). If the ARP / ND related data is ARP / ND response data received by the POD, no processing is performed and the ARP / ND protocol stack in the POD kernel handles it (step S4).

[0106] 3. Agent module in the node:

[0107] If the data being forwarded is ARP / ND request data, and if the IP address and the corresponding relationship between the IP address and the MAC address in the request data have been recorded locally, then ARP / ND response data is directly constructed and responded to the corresponding requester (step S5).

[0108] If the forwarded data is ARP / ND request data, if the local area does not record the IP address in the request data and the correspondence between the IP address and the MAC address, the proxy module needs to actively learn the MAC address corresponding to the IP address in the request data. Therefore, the proxy module constructs and broadcasts the ARP / ND request data (step S6). After receiving the ARP / ND response data from the other end, it updates the local IP address-MAC address correspondence, and constructs ARP / ND response data for the forwarded ARP / ND request data to respond to the original ARP / ND requester (step S7).

[0109] If the forwarded data is ARP / ND announcement data (the ARP / ND announcement data is generally the initial address of the sender of the ARP / ND announcement data, and its IP address-MAC address relationship will be announced in the broadcast domain), the proxy module checks whether the IP address and MAC address of the ARP / ND announcement data are the IP address of the POD of the local K8S system. If it is an IP address within the local K8S system, the announcement data is discarded (the IP address-MAC address of the POD in the K8S system is static information configured through the control plane channel and does not need to be updated) (step S8). If it is not an IP address within the local K8S system, and the IP address in the announcement data does not conflict with the IP address of the POD of the local K8S system, the local IP address-MAC address correspondence is updated (step S9). If it is not an IP address within the local K8S system, and the IP address in the announcement data conflicts with the IP address of the POD of the local K8S system, the response data of the announcement data is constructed to respond to the initiator, informing it of the address conflict (step S10).

[0110] Scenario 2: Update the highly active IP address-MAC address to the POD local ARP table, as shown in Figure 6:

[0111] (1) The eBPF program in each POD counts the target IP addresses whose activity of the data packets sent and received is greater than the preset activity.

[0112] (2) The CNI plug-in on the working node periodically collects the active target IP addresses obtained from the statistics of each POD and updates the target IP addresses to the proxy module. The proxy module determines the active target IP address-target MAC relationship and sends the target IP address-target MAC relationship to the CNI plug-in. The CNI plug-in sends the target IP address-target MAC relationship to the POD, and the POD updates the corresponding static ARP table (i.e., the target IP address-target MAC relationship). In this way, for some frequently accessed target IP addresses, the POD no longer needs to learn and can directly send and receive IP packets, which greatly reduces the first packet data delay and also reduces the broadcast volume in the system.

[0113] (3) The proxy module monitors changes in the active target IP address-target MAC address relationship (such as POD rebirth, etc.), notifies the CNI plug-in, and the CNI plug-in notifies the POD, which promptly updates the local static ARP table.

[0114] In scenario 3, the proxy module supports enabling and disabling functions, as shown in Figure 7:

[0115] 1. The CNI plug-in monitors the status of the proxy module. This monitoring method includes observing the running status of the proxy module and attempting to send ARP / ND requests to the proxy module to obtain the IP address-MAC address relationship. If the proxy module is not running properly or cannot respond to ARP / ND requests, the proxy module status is considered abnormal.

[0116] 2. After the CNI plug-in detects an abnormal proxy module status, it disables the module. This disables the interception function of the eBPF program on all pods of the current working node where the abnormal proxy module is located. This allows the pods of the working node where the abnormal proxy module is located to process and learn ARP / ND packets on their own. At the same time, the local static ARP tables of all pods of the current working node are cleared.

[0117] 3. After the CNI plug-in detects that the abnormal proxy module has returned to normal, it implements the enable function, that is, turns on the interception function of all POD eBPF programs of the current node where the proxy module is located.

[0118] Scenario 4: The proxy module supports the IP address-MAC address aging function:

[0119] 1. For IP addresses and MAC addresses outside the system (non-system IP-MAC), the proxy module records the creation time for each IP address and MAC address outside the system.

[0120] 2. The proxy module periodically checks whether the aging time of the IP address-MAC address outside the system (i.e., the duration from the creation time to the current time) exceeds the preset time. If it exceeds the preset time and the IP address-MAC address outside the system whose aging time exceeds the preset time is not in the active target IP address-target MAC address relationship list, the local IP address-MAC address configuration outside the system whose aging time exceeds the preset time is cleared.

[0121] In summary, the system using the embodiments of the present disclosure has the following technical advantages:

[0122] (1) The broadcast packets caused by a large number of PODs in a working node are terminated in the working node itself, which will not affect the capacity and performance of external infrastructure (such as switches), and can support ultra-large-scale container platforms.

[0123] (2) The local working node responds to the ARP / ND request data, which greatly speeds up the ARP / ND response time for cross-nodes and improves the user's first packet processing performance.

[0124] (3) Customizable policy based on the broadcast suppression function of the characteristic MAC.

[0125] (4) Using Underlay technology, POD can interact with external network devices at L2, with high forwarding performance.

[0126] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0127] In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0128] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present disclosure shall fall within the scope of the present disclosure.

Claims

1. A container cluster management system, the system comprising a plurality of working nodes, the working nodes comprising at least one namespace, the namespace comprising a virtual network interface, the system further comprising: Interception module, forwarding bridge and proxy module; The interception module is arranged at the virtual network interface, and is arranged to monitor and intercept data passing through the virtual network interface, and when the data is non-business data, forward the non-business data to the proxy module for processing through the forwarding bridge, and when the data is business data, do not intercept the business data; The forwarding bridge is arranged between the namespace and the working node, and is arranged to provide communication capability between the namespace and the working node, and forward the non-business data intercepted by the interception module to the proxy module; The proxy module is arranged in the working node and is arranged to process the received non-business data.

2. The system according to claim 1, wherein: The interception module is further configured to forward the first non-business data to the proxy module for processing when the non-business data is first non-business data, and not intercept the second non-business data when the non-business data is second non-business data.

3. The system according to claim 2, wherein: The first non-business data includes first ARP / ND request data, and the second non-business data includes first ARP / ND response data; The proxy module is further configured to process the first ARP / ND request data according to a first correspondence between the first IP address in the first ARP / ND request data and the locally cached IP address-MAC address.

4. The system according to claim 3, wherein: The proxy module is further configured to: In the case where the first MAC address corresponding to the first IP address is queried from the first corresponding relationship, generating corresponding second ARP / ND response data, and sending the second ARP / ND response data to the sending end of the first ARP / ND request data; If the first MAC address corresponding to the first IP address is not found from the first correspondence, the corresponding second ARP / ND request data is generated and broadcast, the first correspondence is updated according to the third ARP / ND response data corresponding to the received second ARP / ND request data, and the fourth ARP / ND response data corresponding to the first ARP / ND request data is generated, and the fourth ARP / ND response data is sent to the sender of the first ARP / ND request data.

5. The system according to claim 2, wherein: The first non-business data also includes ARP / ND announcement data; The proxy module is further configured to process the ARP / ND announcement data according to a second correspondence between a second IP address, a second MAC address in the ARP / ND announcement data and a locally cached IP address-MAC address in the system.

6. The system according to claim 5, wherein: The proxy module is further configured to: When it is determined according to the second corresponding relationship that the second IP address and the second MAC address are IP addresses and MAC addresses in the system, discarding the ARP / ND announcement data; When it is determined according to the second correspondence that the second IP address and the second MAC address are not the IP address and the MAC address in the system, and the second IP address does not conflict with the IP address in the system, updating the third correspondence between the IP address and the MAC address outside the system cached locally according to the second IP address and the second MAC address; When it is determined according to the second corresponding relationship that the second IP address and the second MAC address are not the IP address and MAC address within the system, and that the second IP address conflicts with the IP address within the system, response data corresponding to the ARP / ND announcement data is generated, and the response data corresponding to the ARP / ND announcement data is sent to the sender of the ARP / ND announcement data to notify that the second IP address conflicts with the IP address in the namespace within the system.

7. The system according to claim 1, wherein: The system further includes a node management component, wherein the node management component is disposed in the working node. The interception module is further configured to count the target IP addresses in the data whose activity of sending and receiving is greater than a preset activity, and send the target IP addresses to the node management component; The node management component is configured to send the received target IP address to the proxy module; The proxy module is further configured to determine a fourth correspondence between the target IP address and the target MAC address based on the first correspondence between the target IP address and the locally cached IP address-MAC address, and return the fourth correspondence to the node management component; The node management component is further configured to send the received fourth corresponding relationship to the namespace within the working node.

8. The system according to claim 7, wherein: The proxy module is further configured to monitor the fourth corresponding relationship, and when a change in the fourth corresponding relationship is detected, update the fourth corresponding relationship, and return the updated fourth corresponding relationship to the node management component; The node management component is further configured to send the received updated fourth corresponding relationship to the namespace within the working node.

9. The system according to claim 7, wherein: The node management component is also configured to monitor the status of the proxy module. When it is monitored that the status of the proxy module is abnormal, the interception module is controlled to be closed, and the namespace within the working node is controlled to clear the fourth corresponding relationship. When it is monitored that the proxy module with an abnormal status returns to normal, the interception module is controlled to be turned on.

10. The system according to claim 7, wherein: The proxy module is also configured to monitor the aging time of the IP address-MAC address in the third correspondence between the IP address and the MAC address outside the system that is locally cached. When it is monitored that the aging time exceeds the preset time and the IP address-MAC address whose aging time exceeds the preset time is not in the fourth correspondence, the IP address-MAC address whose aging time exceeds the preset time is cleared from the third correspondence to obtain an updated third correspondence.

11. The system according to claim 1, wherein: The forwarding bridge is further configured to forward the non-business data through the network interface of the working node when the MAC address of the namespace of the working node is the same as the MAC address of the network interface of the working node.

12. The system according to any one of claims 1 to 11, wherein: The system includes a K8S system, and the command space includes a Pod.

13. The system according to claim 12, wherein: The interception module includes an extended Berkeley Packet Filter eBPF.

Images