Multi-cluster load balancing method and device, electronic equipment and storage medium

Abstract

The disclosure provides a multi-cluster load balancing method and device, electronic equipment and computer readable storage medium, relates to the technical field of computers, and can be applied to big data, cloud computing and intelligent supply chain. The method comprises the following steps: selecting one from a plurality of sub-clusters as a master cluster, collecting service endpoint information of the master cluster and the sub-clusters through the master cluster, and generating a cluster internal load balancing rule and a multi-cluster load balancing rule according to the service endpoint information, and forwarding data based on an extended Berkeley packet filter technology; and the embodiment of the disclosure can support multi-cluster services.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a multi-cluster load balancing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] The network proxy component kube-proxy is responsible for providing service discovery and load balancing within the Kubernetes cluster for services. However, the business containers behind the load balancer kube-proxy must be within Kubernetes and do not support services in multiple clusters.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides a multi-cluster load balancing method, apparatus, electronic device, and computer-readable storage medium, which at least to some extent overcomes the problem of not supporting multi-cluster services in related technologies.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, a multi-cluster load balancing method is provided, applied to a node, comprising: selecting one of a plurality of sub-clusters to determine a master cluster; collecting service endpoint information of the master cluster and the sub-clusters through the master cluster; generating cluster rules by the master cluster and the sub-clusters based on the service endpoint information, the cluster rules including: intra-cluster load balancing rules and multi-cluster load balancing rules; and forwarding data based on extended Berkeley packet filtering technology according to the cluster rules.

[0007] In one embodiment of this disclosure, the method further includes: setting a custom resource type to be deployed to the main cluster, and determining the main cluster as a global service resource object.

[0008] In one embodiment of this disclosure, the method further includes: allocating a virtual network address to the primary cluster, wherein the virtual network address includes: an internal cluster address range and a multi-cluster address range.

[0009] In one embodiment of this disclosure, the main cluster distributes the service information in the custom resource type to multiple sub-clusters and collects the backend addresses of the sub-clusters, wherein the service information includes the service endpoint information.

[0010] In one embodiment of this disclosure, collecting service endpoint information of the main cluster and the sub-cluster through the main cluster includes: creating a headless service within the main cluster and the sub-cluster; and obtaining the service endpoint information of the main cluster and the sub-cluster based on the headless service.

[0011] In one embodiment of this disclosure, the method further includes: the main cluster monitoring the headless services of the sub-cluster and the main cluster; and filtering the service endpoint information of the headless services through a tag selector.

[0012] In one embodiment of this disclosure, the method further includes: forwarding data via an external load balancer when configuring the cluster rules fails.

[0013] In one embodiment of this disclosure, selecting one sub-cluster from multiple sub-clusters to determine the primary cluster includes: obtaining performance data of the sub-clusters; and selecting the primary cluster based on the performance data.

[0014] According to another aspect of this disclosure, a multi-cluster load balancing device is also provided, comprising:

[0015] The master cluster selection module selects one from multiple sub-clusters to determine the master cluster;

[0016] The server endpoint collection module collects server endpoint information from the main cluster and the sub-clusters through the main cluster;

[0017] The cluster rule configuration module generates cluster rules for the main cluster and the sub-cluster based on the service endpoint information. The cluster rules include: internal load balancing rules and multi-cluster load balancing rules.

[0018] The data forwarding module forwards data based on the cluster rules and extended Berkeley packet filtering technology.

[0019] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described multi-cluster load balancing methods by executing the executable instructions.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the multi-cluster load balancing method described in any of the preceding claims.

[0021] The multi-cluster load balancing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this disclosure select one of multiple sub-clusters to determine as the main cluster, collect service endpoint information of the main cluster and sub-clusters through the main cluster, generate internal load balancing rules and multi-cluster load balancing rules based on the service endpoint information of the main cluster and sub-clusters, and forward data based on extended Berkeley packet filtering technology, which can support multi-cluster services.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This diagram illustrates a multi-cluster load balancing method according to an embodiment of the present disclosure.

[0025] Figure 2 This invention discloses a flowchart of a method for collecting service endpoint information of multiple sub-clusters according to an embodiment of the present disclosure.

[0026] Figure 3 This diagram illustrates a multi-cluster load balancing method according to an embodiment of the present disclosure.

[0027] Figure 4 This diagram illustrates a control plane schematic of a multi-cluster load balancing embodiment of the present disclosure.

[0028] Figure 5 This diagram illustrates a data plane representation of a multi-cluster load balancing method according to an embodiment of the present disclosure.

[0029] Figure 6 This diagram illustrates a multi-cluster load balancing device according to an embodiment of the present disclosure.

[0030] Figure 7 This diagram illustrates a multi-cluster load balancing system according to an embodiment of the present disclosure.

[0031] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] To facilitate understanding, the following is an explanation of several terms used in this disclosure:

[0035] CRD (Custom Resource Definition) is a built-in resource type in a cluster, that is, a definition of a custom resource, used to describe what a user-defined resource looks like. CR is each value of the CRD.

[0036] A cluster of Kubernetes is essentially a cluster of servers. It can run specific programs on each node of the cluster to manage the containers on that node.

[0037] A service is an abstract concept, a resource object with a virtual network address. It maps a specified port to a virtual network address and forwards requests from clients to one of the backend pod nodes.

[0038] A service endpoint is a list of resource objects in a Kubernetes cluster. It stores a list of pod names and IPs for the corresponding service backend and provides the actual ability to forward requests to the pods.

[0039] etcd is a distributed, highly available, consistent key-value store system written in a language that provides reliable distributed key-value storage, configuration sharing, and service discovery.

[0040] The service VIP (virtual Internet Protocol) is provided by the cluster and acts as a service gateway. All requests are intercepted by the service VIP and then forwarded.

[0041] The purpose of ebpf (extended Berkeley packet filter) is to provide a method for filtering packets and to avoid unnecessary packet copying from kernel space to user space.

[0042] Headless services are logical abstractions and access points for a cluster of backend pods that provide the same service.

[0043] DNS (domain name system) stores the mapping relationship between domain names and IP addresses in a distributed database.

[0044] An API (Application Programming Interface) is a set of predefined functions designed to provide applications and developers with the ability to access a set of routines based on certain software or hardware, without needing to access the source code or understand the details of the internal workings.

[0045] nodeport is a port that is accessible to external systems.

[0046] kube-proxy is a simple network proxy and load balancer.

[0047] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0048] First, this disclosure provides a multi-cluster load balancing method, which can be executed by any electronic device with computing power.

[0049] Figure 1 This invention discloses a flowchart of a multi-cluster load balancing method according to an embodiment of the present invention, as shown below. Figure 1 As shown in the embodiments of this disclosure, the multi-cluster load balancing method is applied to nodes and includes the following steps:

[0050] S102, Select one of the multiple sub-clusters to determine the primary cluster;

[0051] In one embodiment, a custom resource type CRD is set up and deployed to the main cluster, which is then identified as a global service resource object, thereby automating resource management and enabling efficient collection of service endpoint information.

[0052] A Kubernetes cluster is essentially a group of servers that can run specific programs on each node to manage containers within those nodes. The goal is to automate resource management. A CRD (Custom Resource Definition) is a built-in resource type within the cluster, defining a user-defined resource and using it to identify what that resource looks like. The CRD is used to define a global service resource object for the main cluster.

[0053] In one embodiment, performance data of multiple sub-clusters is obtained; a master cluster is selected based on the performance data; the master cluster needs to collect service endpoint information of the master cluster and sub-clusters, monitor headless services of the master cluster and sub-clusters, filter the service endpoint information of headless services through a label selector, and select the sub-cluster with the best performance to determine the master cluster, which can ensure the stability of the multi-cluster load balancing method.

[0054] S104, collects server endpoint information of the main cluster and sub-clusters through the main cluster;

[0055] A service is an abstract concept, a resource object with a virtual network address. It maps a specified port to a virtual network address and forwards client requests to one of the backend pod nodes. The cluster combines the IP address information of the service associated with the pod to form a service endpoint. The service endpoint is a list of resource objects in the Kubernetes cluster that stores the names and IPs of the backend pods corresponding to the service and provides the actual ability to forward requests to the pods. It is stored in etcd to record the access address of the business container corresponding to a service.

[0056] etcd is a distributed, highly available, consistent key-value store system written in a programming language. It provides reliable distributed key-value storage, configuration sharing, and service discovery. etcd can be used to store critical data and implement distributed scheduling.

[0057] In one embodiment, each cluster deploys a first cluster component, which is a network proxy component. The first cluster component can be used to allocate virtual network addresses to the main cluster, create headless services in the main cluster and sub-clusters and obtain service endpoint information of the main cluster and sub-clusters, distribute service information to multiple sub-clusters to collect the backend addresses of the sub-clusters, and filter to obtain the service endpoint information of the headless services.

[0058] S106, The main cluster and sub-clusters generate cluster rules based on the server endpoint information. The cluster rules include: internal load balancing rules and multi-cluster load balancing rules.

[0059] The load balancing rules within a cluster are the load balancing rules for pods within the cluster, while the load balancing rules across multiple clusters are the load balancing rules between multiple clusters.

[0060] In one embodiment, the server endpoint information of the sub-cluster is monitored, and the internal load balancing rules of the cluster are configured.

[0061] In one embodiment, the server endpoint information of the main cluster is monitored, and load balancing rules for multiple clusters are configured.

[0062] In one embodiment, the second cluster component is a network proxy component, and the first and second cluster components are network proxies and load balancers, which are the basic components for the cluster to implement service registration and discovery. The second cluster component is deployed on the nodes of the main cluster and each sub-cluster to listen to the service endpoint information of the sub-cluster and the service endpoint information of the main cluster, and configure cluster rules accordingly. The load is balanced based on the load balancing rules within the cluster and the load balancing rules of the multi-cluster, which can achieve the purpose of optimizing resource utilization, maximizing throughput, minimizing response time, and avoiding overload.

[0063] S108 forwards data based on extended Berkeley packet filtering technology, according to cluster rules.

[0064] The virtual network address (service VIP) is provided by the cluster and acts as a service gateway. All requests are intercepted by the service VIP and then forwarded. Clients inside the cluster can access the underlying pod cluster simply through the virtual network address without needing to know the specific number or address of the pods in the cluster. Even if the pod address changes, it will be masked by the virtual network address.

[0065] The purpose of EBPF (Extended Berkeley Packet Filter) is to provide a method for filtering packets and to avoid unnecessary packet copying from kernel space to user space. An EBPF program is attached to a specified kernel code path, and when that code path is executed, the corresponding EBPF program is executed. The EBPF program is attached to the network socket to perform traffic filtering, traffic classification, and network classifier actions.

[0066] In one embodiment, clients within the cluster access a virtual network address on the main cluster. Based on the virtual network address, the actual node pod is selected according to the EBPF program loaded by the second component and the cluster rules. The virtual network address includes an internal cluster address range and a multi-cluster address range. The internal cluster address range corresponds to the service VIP and port within the cluster, while the multi-cluster address range corresponds to the federated service VIP and port. Zero-latency load balancing from pods within the cluster to services is achieved through the internal service VIP and port, and zero-latency load balancing from pods across multiple clusters to services is achieved through the federated service VIP and port.

[0067] In one embodiment, when configuring cluster rules fails, data is forwarded through an external load balancer. When clients within the cluster access the federated service VIP and port, if configuring cluster rules fails, the second component, which has not loaded the EBPF program, can forward data through the external stateless load balancer via the default route, thus not affecting client access to the service and achieving the effect of disaster recovery and backup.

[0068] In the above embodiments, the main cluster collects service endpoint information from the main cluster and sub-clusters. The main cluster and sub-clusters generate internal load balancing rules and multi-cluster load balancing rules based on the service endpoint information. Data is forwarded based on extended Berkeley packet filtering technology, which can achieve seamless switching of client access to services to pod-to-pod access. However, unlike existing iptables and ipvs technologies that add transformation rules on the kernel netfilter framework, it uses EBPF technology to perform transformation at the data layer, which can efficiently achieve seamless switching of client access to services to pod-to-pod access. It supports services in multiple clusters, so that service functions are not limited to the cluster. When business users deploy load balancing services, they are no longer limited to a single cluster and can make more efficient use of cluster resources.

[0069] Figure 2 This invention discloses a flowchart of a method for collecting service endpoint information of multiple sub-clusters according to an embodiment of the present disclosure, as shown below. Figure 2 As shown in the embodiments of this disclosure, the method for collecting service endpoint information of multiple sub-clusters includes the following steps:

[0070] S202, create headless services within the main cluster and sub-clusters; obtain service endpoint information of the main cluster and sub-clusters based on the headless services.

[0071] Headless services are a logical abstraction and access point for a cluster of backend pods providing the same service. They are a special type of service where `service.spec.cluster IP: None` is specified during definition. During runtime, no virtual network address is assigned. The service's DNS (Domain Name System) can be resolved to return all pod addresses and DNS records. In contrast, a regular service can only return its virtual network address by resolving its DNS. Obtaining service endpoint information from multiple sub-clusters through headless services does not consume internal cluster IP address resources, thus saving resources.

[0072] In one embodiment, a DNS alias is created pointing to the service name to prevent the service name from changing.

[0073] In one embodiment, the main cluster monitors multiple sub-clusters and the headless services of the main cluster. By filtering with a label selector, the service endpoint information of the headless services can be obtained efficiently.

[0074] When a cluster creates a service, it uses the service's tag selector to find the pod and creates an endpoint with the same name as the service. When the pod's IP address changes, the endpoint content also changes. When the service receives a request, it uses the endpoint to find the address to forward the request.

[0075] In one embodiment, for a headless service that defines a label selector, the endpoint controller creates service endpoint information endpoints records in the API (Application Programming Interface) and modifies the DNS configuration to return all pod addresses, through which the backend pods of the service can be directly reached; for example, if a headless service named MySQL is created in the cluster, an endpoint named mysql will be generated. The endpoint is the IP address of the pod associated with the service and the container port provided for this service.

[0076] S204 allocates virtual network addresses to the primary cluster. The virtual network addresses include: internal cluster address range and multi-cluster address range.

[0077] In one embodiment, clients within the cluster can access the underlying pod cluster simply through a virtual network address. Based on the virtual network address allocated to the main cluster and the load balancing rules within the cluster and the load balancing rules across multiple clusters, zero-latency load balancing is achieved for pod-to-service access within the cluster and for pod-to-service access across multiple clusters.

[0078] S206, the main cluster distributes service information from custom resource types to multiple sub-clusters and collects the backend addresses of the sub-clusters.

[0079] In one embodiment, the main cluster uses a custom resource CR in a custom resource type CRD, obtains service information from the CR, distributes the service information to multiple sub-clusters, collects the backend addresses of the sub-clusters, and selects the actual backend pods through the backend addresses.

[0080] In one embodiment, server endpoint information can be added to the service information.

[0081] In the above embodiments, the service access control plane within multiple clusters does not depend on whether there is a federated cluster or external cluster mechanism to provide support. It supports a service multi-cluster zero-latency load balancing method, solves the limitation that services cannot be supported across clusters, thereby freeing business users from their dependence on a single cluster and providing multi-cluster zero-latency load balancing functionality.

[0082] Figure 3 This diagram illustrates a method flowchart for a multi-cluster load balancing method according to an embodiment of the present disclosure, as follows: Figure 3 As shown in the embodiments of this disclosure, the multi-cluster load balancing method includes the following steps:

[0083] S302, Control Plane: Select a master cluster from multiple clusters to create a global service resource object to collect service endpoint information from all clusters. Each cluster obtains the global service information to configure its data plane transformation rules.

[0084] Figure 4 This diagram illustrates a control plane schematic of a multi-cluster load balancing embodiment of the present disclosure, as shown below. Figure 4 As shown, the control plane includes: main cluster 401, first sub-cluster 402, second sub-cluster 403, first sub-cluster node 404, and second sub-cluster node 405;

[0085] Define the service CRD and deploy it to the main cluster 401. Select one main cluster 401 from multiple sub-clusters according to actual needs. Deploy the first cluster component in each cluster and allocate a global virtual network address to the main cluster 401 through the first cluster component. The virtual network address includes: internal cluster address range and multi-cluster address range.

[0086] The main cluster 401 uses the service CRD to create a CR, extracts the service information from the CR and sends it to the associated clusters to collect backend addresses from multiple clusters. Within the cluster where the first cluster component is located, i.e., the main cluster 401, the first sub-cluster 402, and the second sub-cluster 403, headless services are created to collect endpoint information of the cluster. The main cluster 401 listens to the headless services of all clusters, filters the endpoint information of the headless services by label, and statically adds the endpoint information to the service information.

[0087] Each cluster node, namely the first sub-cluster node 404 and the second sub-cluster node 405, deploys a second cluster component, listens to the service of its sub-cluster, configures the load balancing rules within the cluster, listens to the service of the main cluster 401, and configures the load balancing rules for multiple clusters.

[0088] S304, Data Plane: Each cluster is configured according to the cluster rules generated by the control plane, and uses EBPF technology to perform packet conversion on bidirectional traffic of data packets.

[0089] Figure 5 This diagram illustrates a data plane representation of a multi-cluster load balancing scheme in an embodiment of the present disclosure, such as... Figure 5 As shown, the data plane includes: a first cluster 501, a second cluster 502, a first server endpoint 503, a second server endpoint 504, and an external load balancer 505;

[0090] Clients within the cluster access the internal service VIP and port of the first cluster 501, and select the real pod and the first service endpoint 503 through the EBPF program loaded by the component of the second cluster 502, thus achieving zero-latency load balancing from pod to service access within the cluster.

[0091] The Federation Service VIP is the control plane that connects multiple clusters. Clients within a cluster access the Federation Service VIP and port of the second cluster via 502. The EBPF program loaded by the second cluster 502 component selects the real pod and the second service endpoint 504, achieving zero-latency load balancing for multi-cluster pod-to-service access.

[0092] When clients outside the cluster access the federated service VIP and port of the second cluster 502, if the second cluster 502 component has not yet loaded the EBPF program, the data can be forwarded through the external load balancer 505 via the default route, thus not affecting the client's access to the service and achieving the effect of disaster recovery and backup.

[0093] Clients within and outside the cluster can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. The clients installed on different clients may be the same type of application, or clients of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client can also differ; for example, the application client can be a mobile client or a PC client.

[0094] nodeport is a port accessible to external systems. Based on nodeport, an external load balancer (505) is created with the help of the microservice provider cloud-provider, and requests are forwarded to nodeport.

[0095] The main data structure used by ebpf programs is the ebpf map, which is a general-purpose data structure used to transfer data between the kernel or between the kernel and user space. Its name "map" also means that the storage and retrieval of data require the use of a key.

[0096] The `ebpf()` system call is used to create and manage maps. Upon successful creation of a map, it returns the file descriptor associated with that map. Closing the corresponding file descriptor destroys the map. Each map defines four values: type, maximum number of elements, size of the values ​​in bytes, and size of the key in bytes. `ebpf` provides different map types, each offering different characteristics. The actual pod can be selected using the `ebpf map` function.

[0097] The following are some common types:

[0098] ebpf_map_type_device_map: for storing and looking up network device references;

[0099] ebpf_map_type_lpm_trie: a longest-prefix match trie, good for matching IP addresses to a range;

[0100] ebpf_map_type_device_map: for storing and looking up network device references.

[0101] S306, Within the Cluster: Load balancing is achieved based on the internal load balancing rules of the cluster.

[0102] Unlike existing iptables and ipvs technologies that add transformation rules to the kernel netfilter framework, the above embodiments use EBPF technology for transformation at the data layer, which can efficiently achieve seamless switching from client access to service to pod-to-pod access.

[0103] Based on the same inventive concept, this disclosure also provides a multi-cluster load balancing device, as shown in the following embodiments. Since the principle by which this device solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0104] Figure 6 This diagram illustrates a multi-cluster load balancing device according to an embodiment of the present disclosure, such as... Figure 6 As shown, the multi-cluster load balancing device 6 includes: a master cluster selection module 601, a server endpoint collection module 602, a cluster rule configuration module 603, and a data forwarding module 604;

[0105] The master cluster selection module 601 selects one of multiple sub-clusters to determine as the master cluster;

[0106] The server endpoint collection module 602 collects server endpoint information from the main cluster and sub-clusters through the main cluster.

[0107] The cluster rule configuration module 603 configures cluster rules for the main cluster and sub-clusters based on the service endpoint information. The cluster rules include: internal load balancing rules and load balancing rules for multiple clusters.

[0108] The data forwarding module 604 forwards data based on extended Berkeley packet filtering technology according to cluster rules.

[0109] In the above embodiments, the main cluster collects service endpoint information from the main cluster and sub-clusters. The main cluster and sub-clusters generate internal load balancing rules and multi-cluster load balancing rules based on the service endpoint information. Data is forwarded based on extended Berkeley packet filtering technology, which can achieve seamless switching of client access to services to pod-to-pod access. However, unlike existing iptables and ipvs technologies that add transformation rules on the kernel netfilter framework, it uses EBPF technology to perform transformation at the data layer, which can efficiently achieve seamless switching of client access to services to pod-to-pod access. It supports services in multiple clusters, so that service functions are not limited to the cluster. When business users deploy load balancing services, they are no longer limited to a single cluster and can make more efficient use of cluster resources.

[0110] Based on the same inventive concept, this disclosure also provides a multi-cluster load balancing system, as shown in the following embodiments. Since the principle by which this system embodiment solves the problem is similar to that of the above method embodiments, the implementation of this system embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0111] Figure 7 The diagram illustrates a multi-cluster load balancing system according to an embodiment of this disclosure, using the example of two different services accessing pods in two clusters for illustration.

[0112] A service is a service abstraction of a group of containers, which is equivalent to a load balancer for a group of containers, responsible for distributing requests to the corresponding business containers; kube-proxy is a simple network proxy and load balancer.

[0113] Two different services, namely service 71 and service 72, can achieve seamless switching from client access to pod access based on multi-cluster load balancing;

[0114] That is, the first service 71 can access the first cluster node 2 712 and the first cluster node 4 714 of the first cluster 73 and the second cluster node 2 722 and the second cluster node 4 724 of the second cluster 74 based on the cluster rules.

[0115] The second service 72 can access the first cluster node 711 and the first cluster node 713 of the first cluster 73, as well as the second cluster node 721 and the second cluster node 723 of the second cluster 74, based on cluster rules, thus solving the problem that the service function is limited to the cluster in related technologies.

[0116] In related technologies, kube-proxy in a Kubernetes cluster supports three modes: userspace, iptables, and ipvs.

[0117] The userspace mode supported by kube-proxy is deprecated;

[0118] kube-proxy supports the following iptables mode: iptables is dynamically managed by kube-proxy, and kube-proxy is no longer responsible for forwarding. The direction of data packets is completely determined by iptables rules. As the number of services increases, the number of iptables rules will continue to increase, causing the kernel to be very busy. For example, 2 services and 8 business containers will have 34 iptables rules. As the number of svcs and pods in the cluster increases significantly, the number of rules in iptables will expand rapidly, leading to performance degradation. In some extreme cases, rules may even be lost. Such failures are difficult to reproduce and troubleshoot.

[0119] kube-proxy supports the IPVS mode: it offers a significant performance improvement over iptables, but IPVS cannot provide features such as packet filtering, address masquerading, and SNAT. Therefore, in some scenarios, it still needs to be used in conjunction with iptables, and the impact on performance is self-evident.

[0120] In the above embodiments, data forwarding based on extended Berkeley packet filtering technology enables seamless switching from client access to pod-to-pod access. However, unlike existing iptables and ipvs technologies that add transformation rules to the kernel netfilter framework, EBPF technology is used for transformation at the data layer, which can efficiently achieve seamless switching from client access to pod-to-pod access. It supports services in multiple clusters, so that service functionality is not limited to the cluster. When deploying load balancing services, businesses are no longer limited to a single cluster and can utilize cluster resources more efficiently.

[0121] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0122] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0123] like Figure 8 As shown, the electronic device 800 is manifested in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0124] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0125] For example, the processing unit 810 may perform the following steps in the above method embodiment: select one from multiple sub-clusters to determine the main cluster; collect the service endpoint information of the main cluster and sub-clusters through the main cluster; configure cluster rules for the main cluster and sub-clusters according to the service endpoint information, the cluster rules including: internal cluster load balancing rules and multi-cluster load balancing rules; and forward data based on extended Berkeley packet filtering technology according to the cluster rules.

[0126] For example, the processing unit 810 can execute the following steps in the above method embodiment: create headless services in the main cluster and sub-clusters; obtain service endpoint information of the main cluster and sub-clusters based on the headless services; allocate virtual network addresses to the main cluster, the virtual network addresses including: internal cluster address range and multi-cluster address range; the main cluster distributes service information in the custom resource type to multiple sub-clusters and collects the backend addresses of the sub-clusters.

[0127] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0128] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0129] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0130] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0132] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0133] For example, when the program product in this embodiment is executed by the processor, it implements the following steps: selecting one of multiple sub-clusters to determine as the main cluster; collecting server endpoint information of the main cluster and sub-clusters through the main cluster; configuring cluster rules for the main cluster and sub-clusters according to the server endpoint information, the cluster rules including: internal cluster load balancing rules and multi-cluster load balancing rules; and forwarding data based on extended Berkeley packet filtering technology according to the cluster rules.

[0134] For example, when the program product in this embodiment is executed by the processor, it implements the following steps: Control plane: Select a master cluster from multiple clusters to create a global service resource object to collect service endpoint information from all clusters. Each cluster obtains the global service information to configure its data plane transformation rules; Data plane: Each cluster configures according to the cluster rules generated by the control plane and applies EBPF technology to perform packet transformation on bidirectional data packet traffic; Intra-cluster: Load balancing is achieved based on the intra-cluster load balancing rules.

[0135] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0137] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0138] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0139] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0140] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0141] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0142] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A multi-cluster load balancing method, characterized in that, Applied to nodes, including: Select one of the multiple sub-clusters to determine the primary cluster; wherein, the first cluster component is deployed in each sub-cluster cluster. The service endpoint information of the main cluster and the sub-clusters is collected through the main cluster; The main cluster and the sub-cluster generate cluster rules based on the service endpoint information. The cluster rules include: internal cluster load balancing rules and multi-cluster load balancing rules. Data is forwarded based on the extended Berkeley packet filtering technology, according to the cluster rules. The method further includes: The first cluster component allocates a virtual network address to the main cluster, and the virtual network address includes: an internal cluster address range and a multi-cluster address range; Clients within the cluster access the virtual network address on the main cluster and select the real node pod based on the virtual network address, the extended Berkeley packet filter, and cluster rules. The method includes: Obtain the performance data of the sub-cluster; The primary cluster is selected based on the performance data.

2. The multi-cluster load balancing method according to claim 1, characterized in that, Also includes: Configure a custom resource type to be deployed to the main cluster, and designate the main cluster as a global service resource object.

3. The multi-cluster load balancing method according to claim 2, characterized in that, Also includes: The main cluster distributes the service information in the custom resource type to multiple sub-clusters and collects the backend addresses of the sub-clusters. The service information includes the service endpoint information.

4. The multi-cluster load balancing method according to claim 3, characterized in that, The step of collecting service endpoint information of the main cluster and sub-clusters through the main cluster includes: Create headless services within the main cluster and the sub-cluster; The service endpoint information of the main cluster and the sub-cluster is obtained based on the headless service.

5. The multi-cluster load balancing method according to claim 4, characterized in that, Also includes: The main cluster monitors the headless services of the sub-cluster and the main cluster; The service endpoint information of the headless service is filtered using a tag selector.

6. The multi-cluster load balancing method according to claim 1, characterized in that, Also includes: When configuring the cluster rules fails, data is forwarded through an external load balancer.

7. A multi-cluster load balancing device, characterized in that, include: The master cluster selection module selects one from multiple sub-clusters to determine the master cluster; each cluster deploys the first cluster component. The server endpoint collection module collects server endpoint information from the main cluster and the sub-clusters through the main cluster; The cluster rule configuration module generates cluster rules for the main cluster and the sub-cluster based on the service endpoint information. The cluster rules include: internal load balancing rules and multi-cluster load balancing rules. The data forwarding module forwards data based on the cluster rules and extended Berkeley packet filtering technology. The multi-cluster load balancing device is also used to allocate virtual network addresses to the main cluster through the first cluster component, wherein the virtual network addresses include: internal cluster address range and multi-cluster address range; Clients within the cluster access the virtual network address on the main cluster and select the real node pod based on the virtual network address, the extended Berkeley packet filter, and cluster rules. The primary cluster selection module is also used to obtain the performance data of the sub-clusters and select the primary cluster based on the performance data.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the multi-cluster load balancing method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-cluster load balancing method according to any one of claims 1 to 6.

Images