Method, device and storage medium for deploying distributed container orchestration management cluster

Abstract

The present application discloses a method, device and storage medium for deploying a distributed container orchestration management cluster, and relates to the field of cloud technology. The method comprises: configuring a working node according to the physical IP address of any master node in the distributed container orchestration management cluster to be deployed; after adding the configured working node to the distributed container orchestration management cluster, automatically generating virtual routing rules on the worker node through the distributed container orchestration management cluster; using LVS rules to change the configuration of components in the working node, and restarting the components. After configuring a working node using the physical IP of any master node, the present application adds the working node to the cluster, generates virtual routing rules on the working node, and then uses the virtual routing rules to change the configuration of the kubelet component and the kube-proxy component in the working node, and after restarting the components in the working node, realizes the high availability of the master node to the working node. The method does not rely on external components such as ansible, haproxy and keepalived, and does not require additional resource consumption.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of cloud technology, and in particular to a method, device, electronic device, and storage medium for deploying a distributed container orchestration management cluster. Background Art

[0002] With the rapid development of cloud computing and big data, new technology frameworks are constantly emerging, and Kubernetes (k8s) has emerged as the answer. It is a brand-new distributed architecture based on container technology, built on Docker technology, and provides containerized applications with a rich variety of functions, including resource scheduling, deployment and operation, service discovery, and scaling. K8s is also an open platform, allowing community users to directly participate in application and development frameworks. A key feature of k8s is automation, which includes automatic deployment, automatic restart, automatic replication, and automatic scaling / expansion.

[0003] In terms of cluster management, Kubernetes divides the machines in a cluster into a master node and a group of worker nodes. The master node runs a set of cluster management-related processes, implementing management capabilities such as resource management, pod scheduling, elastic scaling, and security controls for the entire cluster. Nodes, as the working nodes in the cluster, run the actual applications. The smallest operating unit managed by Kubernetes on a node is the pod. Nodes run the Kubernetes kubelet and kube-proxy service processes, which are responsible for pod creation, startup, detection, restart, and termination, as well as implementing a software-based load balancer.

[0004] In existing technologies, high availability deployment of the Master node in a Kubernetes cluster generally relies on external components such as Ansible, haproxy, and keepalived. However, using these external components to achieve high availability of the Master in a Kubernetes cluster has the following problems:

[0005] 1. Because a virtual IP address is required, this is not possible in some private deployment scenarios.

[0006] 2. IP drift during operational failures can add complexity, making it more difficult to troubleshoot and locate unexpected issues.

[0007] 3. Since each Master node needs to run and detect additional HAProxy and Keepalived services, the management and operation burden of the entire cluster is increased. Summary of the Invention

[0008] The present invention provides a method, device, and storage medium for deploying a distributed container orchestration management cluster, enabling high-availability deployment of Master nodes without relying on external components such as Ansible, haproxy, and keepalived. The technical solution is as follows:

[0009] In one aspect, a method for deploying a distributed container orchestration management cluster is provided, the method comprising:

[0010] Configure the worker node based on the physical IP address of any master node in the distributed container orchestration management cluster to be deployed;

[0011] After adding the configured working node to the distributed container orchestration management cluster, a virtual routing rule is automatically generated on the working node.

[0012] The virtual routing rule includes a mapping relationship between a virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster;

[0013] The configuration of the component of the working node is changed using the virtual routing rule, and the component of the working node is restarted.

[0014] In another possible implementation, the distributed container orchestration management cluster to be deployed is a Kubernetes cluster, and configuring a working node according to the physical IP address of any master node in the distributed container orchestration management cluster to be deployed includes:

[0015] Configure the cluster server of the kubelet component and kube-proxy component of the worker node to the physical IP address of any of the Master nodes;

[0016] Set the forwarding mode of the kube-proxy component of the working node to ipvs mode and the scheduling algorithm to wrr algorithm.

[0017] In another possible implementation, the virtual routing rule is a Linux virtual server LVS rule, and the virtual IP address in the LVS rule is the clusterIP of the kubernetes service of the Kubernetes cluster, wherein the cluster IP of the kubernetes service is the first address in the Cluster IP address pool.

[0018] In yet another possible implementation, using the virtual routing rule to change the configuration of a component in the working node includes:

[0019] Change the cluster server of the kubelet component and kube-proxy component of the worker node to the cluster IP of the Kubernetes service in the LVS rule.

[0020] In yet another possible implementation, automatically generating a virtual routing rule on the working node includes:

[0021] The kube-proxy component on the working node obtains the change information of the services and endpoints created by the Kubernetes cluster in real time through the detection interface of the Kubernetes cluster, and automatically generates and maintains the LVS rules on the working node.

[0022] In yet another possible implementation, the method further includes:

[0023] When a failure is detected in any of the Master nodes in the Kubernetes cluster, the physical IP address of the failed Master node is deleted from the virtual routing rules through the Kubernetes cluster to update the virtual routing rules on all working nodes in the Kubernetes cluster.

[0024] In another aspect, a device for deploying a distributed container orchestration management cluster is provided, the device comprising:

[0025] The configuration module is used to configure the worker node according to the physical IP address of any master node in the distributed container orchestration management cluster to be deployed;

[0026] A generation module is used to automatically generate virtual routing rules on the configured working node after adding it to the distributed container orchestration management cluster.

[0027] The virtual routing rule includes a mapping relationship between a virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster;

[0028] A changing module is used to change the configuration of the components of the working node by using the virtual routing rules and restart the components of the working node.

[0029] In another possible implementation, the distributed container orchestration management cluster to be deployed is a Kubernetes cluster, and the configuration module is specifically configured to:

[0030] Configure the cluster server of the kubelet component and kube-proxy component of the worker node to the physical IP address of any of the Master nodes;

[0031] Set the forwarding mode of the kube-proxy component of the working node to ipvs mode and the scheduling algorithm to wrr algorithm.

[0032] In another possible implementation, the virtual routing rule is a Linux virtual server LVS rule, and the virtual IP address in the LVS rule is the cluster IP of the kubernetes service of the Kubernetes cluster, wherein the cluster IP of the kubernetes service is the first address in the Cluster IP address pool.

[0033] In another possible implementation, the changing module is specifically configured to change the cluster server of the kubelet component and the kube-proxy component of the working node to the cluster IP of the kubernetesservice in the LVS rule.

[0034] In another possible implementation, the generation module is specifically used to obtain change information of services and endpoints created by the Kubernetes cluster in real time through the kube-proxy component on the working node through the detection interface of the Kubernetes cluster, and automatically generate and maintain the LVS rules on the working node.

[0035] In yet another possible implementation, the method further includes:

[0036] An update module is used to delete the physical IP address of the failed Master node from the virtual routing rules through the Kubernetes cluster when a failure of any of the Master nodes in the Kubernetes cluster is detected, so as to update the virtual routing rules on all working nodes in the Kubernetes cluster.

[0037] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for deploying a distributed container orchestration management cluster as described above.

[0038] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the method for deploying a distributed container orchestration management cluster as described above.

[0039] On the other hand, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the method for deploying a distributed container orchestration management cluster as described above.

[0040] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0041] After configuring a worker node using the physical IP address of any Master node, the worker node is added to the cluster, a virtual routing rule is generated on the worker node, and the virtual routing rule is used to modify the configuration of the kubelet and kube-proxy components in the worker node. After restarting the components in the worker node, high availability of the Master node to the worker nodes is achieved. This method does not rely on external components such as Ansible, haproxy, and keepalived, and does not require additional resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 This diagram shows the general architecture of a high-availability deployment solution for a Kubernetes cluster.

[0044] Figure 2 This diagram shows the framework for achieving high availability of the Master node using HAProxy and Keepalived in the existing solution.

[0045] Figure 3 A schematic flow chart of a method for deploying a distributed container orchestration management cluster provided by an exemplary embodiment of the present application is shown;

[0046] Figure 4 A schematic flow chart of a method for deploying a distributed container orchestration management cluster provided by another exemplary embodiment of the present application is shown;

[0047] Figure 5This figure shows a framework diagram for using LVS as a reverse proxy to achieve high availability of the Master node in this application;

[0048] Figure 6 A schematic flow chart of a method for deploying a distributed container orchestration management cluster provided by another exemplary embodiment of the present application is shown;

[0049] Figure 7 A schematic diagram of a structure of an apparatus for deploying a distributed container orchestration management cluster provided by an exemplary embodiment of the present application is shown;

[0050] Figure 8 A block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0052] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0053] The following describes an embodiment of the present application in detail. Examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0054] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0055] In order to better understand and illustrate the solutions of the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below:

[0056] Cloud technology: A general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. This technology forms a resource pool, enabling flexible and convenient on-demand access. Cloud computing will become a crucial enabler. Backend services for technical network systems, such as those for video sites, image sites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, requiring robust system support across all industries, a reality only possible through cloud computing.

[0057] Cloud computing refers to the delivery and usage model of IT infrastructure, enabling on-demand, scalable access to required resources over the internet. In a broader sense, cloud computing refers to the delivery and usage model of services, enabling on-demand, scalable access to required services over the internet. These services can be IT-related, software-related, internet-related, or other services. Cloud computing is the product of the convergence of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0058] Kubernetes (abbreviated as k8s): A new open source system for automating the deployment, scaling, and management of containerized applications. It is an open source version of Borg. Kubernetes is based on container technology and adopts a distributed architecture. It features comprehensive cluster management capabilities and multi-layered security. It provides containerized service discovery and load balancing, automated updates and rollbacks, and automatic scheduling and repair. It is a comprehensive, one-stop distributed system development and support platform.

[0059] Linux Virtual Server (LVS): LVS uses the Linux operating system as a load balancer to create a highly scalable and available server cluster. To end users, the server architecture is completely transparent. Users interact with the cluster as if they were interacting with a single, high-performance virtual server.

[0060] In terms of cluster management, Kubernetes divides the machines in the cluster into a Master node and a group of Worker nodes. The Master node acts as the central control center. The kube-apiserver, kube-controller-manager, and kube-scheduler components running on it maintain the healthy working state of the entire cluster by continuously communicating with the kubelet and kube-proxy on the Worker nodes. If the Master service cannot access a Node, it will be marked as unavailable and no new Pods will be scheduled to it. However, the Master itself requires additional testing to ensure that it does not become a single point of failure in the cluster, so the Master service also needs to be deployed with high availability.

[0061] like Figure 1 Each Master node in a Kubernetes cluster contains three components: the API server, the scheduler, and the controller-manager. The API server, in turn, relies on storage (etcd) for data storage. There are two deployment topologies for etcd: stacking it with the API server on the Master node, or directly using an external cluster. The high availability of the Master node is reflected in the high availability of these three components and etcd:

[0062] 1. etcd is a distributed key-value storage system that uses the HTTP protocol. The high availability of etcd is guaranteed by its own cluster.

[0063] 2. For the controller and scheduler, high availability not only requires launching multiple instances, but also requires these instances to elect a leader instance to ensure that only one instance can read and write cluster state information at a time, avoiding synchronization and consistency issues. Kubernetes implements this election mechanism using a lease lock. By setting --leader-elect=true in the startup parameters of each controller and scheduler instance, you can ensure that only one instance can modify cluster information at a time.

[0064] 3. After deploying multiple instances of the API server, accessing any instance will work normally. External services are generally provided through a unified load balancer.

[0065] As can be seen, the key to achieving high availability of the Master node in a Kubernetes cluster lies in ensuring high availability of the API server. This is typically achieved by deploying a load balancer. Load balancing varies across platforms. Public cloud providers like GCE, AWS, and Alibaba Cloud offer pre-existing load balancers. For on-premises clusters, load balancing is typically achieved through software. The most common solution is HAProxy and Keepalived, with HAProxy responsible for load balancing and Keepalived responsible for detecting and failing over HAProxy.

[0066] In a stacked etcd deployment, the solution architecture using HAProxy and Keepalived as the load balancer layer is as follows Figure 2 Keeplived's primary function is to provide VIP for HAProxy, providing a primary / backup relationship between the three HAProxy instances to minimize the impact on services if one of them fails. HAProxy, on the other hand, provides reverse proxy functionality, achieving load balancing across multiple API-server instances.

[0067] However, using HAProxy and Keepalived to implement Kubernetes master high availability has the following problems:

[0068] 1. Because a virtual IP address is required, this is not possible in some private deployment scenarios.

[0069] 2. IP drift during operational failures can add complexity, making it more difficult to troubleshoot and locate unexpected issues.

[0070] 3. Since each master node needs to run and detect additional HAProxy and Keepalived services, the management and operation burden of the entire cluster is increased.

[0071] The inventors of this application discovered during their research on "solving the problem of high-availability deployment of master nodes" that they could use components such as envoy and nginx to implement local reverse proxy on worker nodes, thus achieving high availability of the master node for the worker nodes. However, using these components would have the following disadvantages:

[0072] 1. The traffic from the worker node to the master node is forwarded to the user-mode process, resulting in limited load balancing capabilities.

[0073] 2. A process needs to be run on each node, consuming more resources;

[0074] 3. Additional daemons or scripts are required to detect and dynamically update the load balancing rules of envoy or nginx.

[0075] Therefore, in order to solve the above problems, this application proposes a method that does not rely on external components and only uses LVS technology to configure local kernel-mode load balancing on the working node, combined with the endpoints detection and automatic update mechanism of the service created by Kubernetes itself to achieve high-availability deployment of the master node.

[0076] Specifically, the method for deploying a Kubernetes cluster proposed in an embodiment of this application utilizes the LVS rules generated and maintained by the Kubernetes cluster itself when kube-proxy is in IPVS mode to act as a reverse proxy on the worker nodes. This allows high availability of the master's API-server service to all worker nodes without introducing additional components. Compared with existing solutions, the method proposed in this application has the following advantages:

[0077] 1. Since the Kubernetes cluster itself generates LVS rules, it is not subject to the restrictions of private deployment conditions;

[0078] 2. Once an operational failure occurs, the Kubernetes cluster automatically maintains LVS rules, eliminating the need for IP drift and making troubleshooting and location easier.

[0079] 3. No need to increase additional resource consumption.

[0080] Therefore, the method of the present application achieves high availability of the master node for the working nodes at extremely low cost and maintenance cost.

[0081] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 3 An exemplary embodiment of the present application provides a method 100 for deploying a distributed container orchestration management cluster. The method 100 includes:

[0083] 101. Configure a working node based on the physical IP address of any Master node in the distributed container orchestration management cluster to be deployed;

[0084] 102. After adding the configured working node to the distributed container orchestration management cluster, automatically generate virtual routing rules on the working node.

[0085] The virtual routing rule includes the mapping relationship between the virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster.

[0086] 103. Use the virtual routing rule to change the configuration of the component of the working node, and restart the component of the working node.

[0087] That is to say, in this embodiment, after configuring a working node using the physical IP of any Master node, the working node is added to the cluster, a virtual routing rule is generated on the working node, and then the configuration of the kubelet component and the kube-proxy component in the working node is changed using the virtual routing rule. After restarting the components in the working node, high availability of the Master node to the working node is achieved.

[0088] Kube-proxy is a core component of Kubernetes, deployed on every worker node. It is crucial for implementing the communication and load balancing mechanisms of Kubernetes services. Kube-proxy is responsible for creating proxy services for pods, obtaining all server information from the API server, and creating proxy services based on this information. It routes and forwards requests from servers to pods, thus implementing a virtual forwarding network at the Kubernetes level.

[0089] In Kubernetes, a group of pods providing the same service can be abstracted into a single service. This service is exposed through a unified entry point. Each service has a virtual IP address (VIP) and port number for client access. Kube-proxy exists on each Node and is primarily used to implement service functionality. Specifically, it enables client pods within the cluster to access services, and hosts outside the cluster to access services through methods such as NodePort.

[0090] In some embodiments, the distributed container orchestration management cluster to be deployed is a Kubernetes cluster, and step 101 may specifically include:

[0091] Configure the cluster server of the kubelet component and kube-proxy component of the worker node to the physical IP address of any of the Master nodes;

[0092] Set the forwarding mode of the kube-proxy component of the working node to ipvs mode and the scheduling algorithm to wrr algorithm.

[0093] Specifically, in this embodiment, after the configured working node is added to the kubernetes cluster, the kube-proxy component on the working node will use the cluster's detection interface to obtain real-time service and endpoint change information, thereby automatically generating and maintaining virtual routing rules on this working node.

[0094] The virtual routing rule is a Linux virtual server LVS rule, and the virtual IP address in the LVS rule is the cluster IP of the kubernetes service of the Kubernetes cluster, wherein the cluster IP of the kubernetes service is the first address in the Cluster IP address pool.

[0095] Since the IPVS mode uses a hash table as the basic data structure and works in kernel space, when the number of services reaches a certain scale, the speed advantage of hash table lookup will become apparent, thereby improving the service performance.

[0096] Kube-proxy is responsible for providing service discovery and load balancing within the cluster. It runs on every worker node and acts as a Pod network proxy. It periodically retrieves service information from the etcd service to implement corresponding policies, maintain network rules, and perform Layer 4 load balancing. Load balancing for microservices in a Kubernetes cluster is implemented by Kube-proxy, an internal load balancer and distributed proxy server. Each Kubernetes node has a Kube-proxy, providing scalability advantages. The more nodes that need to access a service, the more Kube-proxy instances are deployed to provide load balancing capabilities, resulting in a higher number of highly available nodes.

[0097] A service is an abstraction of a group of pods, acting as a load balancer for a group of pods, responsible for distributing requests to the corresponding pods. The service provides an IP address for this load balancer, typically called a cluster IP address.

[0098] In this embodiment, when the kube-proxy component's forwarding mode is IPVS, Kube-Proxy detects the Kubernetes Service object and Endpoints, calls the Netlink interface to create IP virtual server (IPVS) rules accordingly, and regularly synchronizes these rules with the Kubernetes Service and Endpoints objects to ensure the IPVS state is consistent with expectations. When accessing the service, traffic is redirected to one of the backend pods.

[0099] An Endpoint is a resource object in a Kubernetes cluster, stored in etcd, that records the access addresses of all pods corresponding to a service. Only when a service is configured with a selector will the Endpoint controller automatically create the corresponding Endpoint object; otherwise, no Endpoint object is generated.

[0100] For example, when a service named hello is created in a k8s cluster, an Endpoint object with the same name is generated. Endpoints are the IP address and port of the pod associated with the service.

[0101] A Service consists of a set of backend Pods. These Pods are exposed through endpoints. The Service Selector is continuously evaluated, and the evaluation results are POSTed to an Endpoint object named Service-hello. When a Pod terminates, it is automatically removed from the Endpoint, and new Pods that match the Service Selector are automatically added to the Endpoint. Inspecting the Endpoint, notice that the IP address is the same as the created Pod.

[0102] Endpoints is a collection of endpoints that implement actual services.

[0103] When creating a Service, Kubernetes uses the Service's label selector to find the Pod and then creates an Endpoints object with the same name as the Service. When the Pod's address changes, the Endpoints object also changes. When the Service receives a request, it uses the Endpoints object to find the destination address to which the request should be forwarded.

[0104] In this embodiment, the scheduling algorithm is set to WRR, namely weighted round-robin scheduling, to achieve load balancing. The principle of the weighted round-robin algorithm is to assign different weights to each server according to its different processing capabilities so that it can accept service requests with corresponding weights.

[0105] For every 7 client requests received, 1 of them will be forwarded to backend a, 2 of them will be forwarded to backend b, and 4 of them will be forwarded to backend c.

[0106] The weighted round-robin algorithm generates a server sequence. Each time a request arrives, the next server in the sequence is selected to handle it. For example, in the example above, the weighted round-robin algorithm generates the sequence {c, c, b, c, a, b, c}. Thus, for every seven client requests received, one is forwarded to backend a, two to backend b, and four to backend c. For the eighth request, round-robin restarts at the beginning of the sequence.

[0107] In summary, the weighted round-robin algorithm generates a server sequence consisting of n servers, where n is the sum of the weights of all servers. Each server appears a number of times in the sequence equal to its weight. Furthermore, the distribution of servers in the generated sequence should be as even as possible.

[0108] In this embodiment, the forwarding mode of the kube-prox component of the working node is set to ipvs mode, and the scheduling algorithm is set to wrr algorithm. In this case, when the client accesses the service, the kube-proxy component will poll the Endpoints in sequence in the kernel state with a shorter delay, and finally direct it according to the weight of each endpoint.

[0109] Specifically, in this embodiment, step 103 may include:

[0110] Change the cluster server of the kubelet component and kube-proxy component of the worker node to the cluster IP of the kubernetes service in the LVS rule.

[0111] Specifically, the cluster IP of the Kubernetes service is the first address in the Cluster IP address pool.

[0112] In some embodiments, as Figure 4 As shown, the method 100 may further include:

[0113] 104. When it is detected that any of the Master nodes in the Kubernetes cluster fails, the physical IP address of the failed Master node is deleted from the LVS rule through the Kubernetes cluster to update the virtual routing rules on all working nodes in the Kubernetes cluster.

[0114] That is to say, when a Master node fails, the physical IP address of the node can be accurately obtained, and the physical IP address of the node can be deleted from the virtual routing rules, while informing other nodes that the node is unavailable.

[0115] In order to describe in more detail the method for deploying a distributed container orchestration management cluster provided by this application, Figure 5 and Figure 6 The example shown takes the deployment of a Kubernetes cluster as an example to describe in detail the method provided in the embodiments of the present application.

[0116] by Figure 5For example, the three initialized Master nodes are Master1-3, and an LVS rule with the address and port 10.254.0.1:443 is configured on the working node. This address and port uses the TCP protocol, the wrr scheduling algorithm, and the NAT mode. The addresses of the three real servers point to the api-server services of the three Master nodes.

[0117] The address 10.254.0.1 is the address of the Kubernetes service itself. The api-server of the Kubernetes cluster is also a service named kubernetes. The Cluster IP address of the service is the first address in the Cluster IP address pool. The service port is HTTPS port 443. Figure 3 shown.

[0118] If you set the proxy mode of kube-proxy to ipvs when deploying a worker node, Kubernetes will automatically generate the same proxy mode as Figure 3 Kubernetes also clears all user-created LVS rules for the Kubernetes Service. If any replica of the API server fails (i.e., if any of Master1-3 nodes fails), Kubernetes updates the endpoints list of the Kubernetes Service on all worker nodes, thereby updating the LVS rules on each worker node.

[0119] Therefore, without any other components, by leveraging this feature of Kubernetes, after a worker node joins the cluster and successfully creates a Kubernetes Service rule on this node, restart the worker node's kube-proxy and kubelet components, and then set its server address configuration to the address of the Kubernetes Service itself (that is, the first address in the Cluster IP address pool, for example: https: / / 10.254.0.1:443), you can achieve high availability of Master1-3 nodes for the worker node.

[0120] The specific high availability deployment steps are as follows:

[0121] 1. First, deploy etcd in a cluster: it can be deployed on a shared machine with the Master node, or deployed outside the Kubernetes cluster machine;

[0122] The specific deployment process of etcd is implemented using the deployment method in the existing technology. For the sake of brevity, it is not repeated here.

[0123] 2. High-availability deployment of the Master node: The Master's API server, controller, and scheduler are deployed as a unit, similar to the typical deployment configuration of an etcd cluster. Install the Master service on at least three servers. These three services can be deployed as binary systems managed by systemd, or deployed as containerized static pods using kubelet for detection and automatic restart. Ensure that leader election is enabled on each Master node's controller and scheduler, and that the API server is functioning properly, before proceeding to the next step.

[0124] 3. The working node joins the cluster:

[0125] Configure a worker node using the physical IP address of any master node in the cluster, and then add the worker node to the cluster. Figure 6 As shown in the figure, the process of adding a worker node to the cluster includes:

[0126] 11. Configure the kubelet of the worker node and the cluster server of the kube-proxy component to the physical IP address of any Master node in the cluster;

[0127] 12. Set the forwarding mode of the kube-proxy of the worker node to ipvs, the scheduling algorithm to wrr, and then add the worker node to the cluster.

[0128] 13. After adding the configured worker node to the cluster, a Service named kubernetes will be automatically created by kube-proxy. The kube-proxy component of the worker node obtains the change information of the services and endpoints created by the cluster in real time through the detection interface of the cluster, and automatically generates and maintains LVS rules on the worker node. The LVS rule includes the mapping relationship between the virtual IP address and the physical IP address of all Master nodes in the cluster. The virtual IP address in the LVS rule is the cluster IP of the kubernetes service of the cluster, and the cluster IP of the kubernetesservice is the first address in the cluster IP address pool. For example: Figure 5 https: / / 10.254.0.1:443 in.

[0129] 4. The worker node uses LVS reverse proxy to achieve high availability of access to the api-server service: Specifically, use the LVS rules generated in step 13 to change the configuration of the component of the worker node and restart the component of the worker node to achieve high availability of access to the api-server service on the Master for all worker nodes.

[0130] The specific process is as follows Figure 6 As shown:

[0131] 14. Change the cluster server in the configuration of the kubelet and kube-proxy components of the worker node to the cluster IP of the Kubernetes service, and restart the kubelet and kube-proxy services. This ensures high availability of the master's api-server service for all worker nodes.

[0132] 15. When a failure is detected on any Master node in the cluster, the physical IP address of the failed Master node is deleted from the LVS rules through the cluster, so that the LVS rules on all working nodes in the cluster can be updated.

[0133] When the deployment method of the distributed container orchestration management cluster proposed in this application is applied to a Kubernetes cluster, the LVS rules generated and maintained by the Kubernetes cluster itself in ipvs mode by kube-proxy can be used as a reverse proxy on the working nodes. This can achieve high availability of the api-server service on the Master for all working nodes without introducing additional components. Compared with existing solutions, the method proposed in this application is not restricted by private deployment conditions, does not require IP drift, is easier to troubleshoot and locate, does not require additional resource consumption, and achieves high availability of the Master node for the working nodes at a very low cost and maintenance cost.

[0134] Based on the same inventive concept, the embodiment of the present application also provides a device for deploying a distributed container orchestration management cluster, the structural diagram of which is shown in FIG. Figure 7 As shown, the apparatus 200 includes: a configuration module 201, a generation module 202 and a modification module 203, wherein:

[0135] Configuration module 201, configured to configure a working node according to the physical IP address of any master node in the distributed container orchestration management cluster to be deployed;

[0136] The generation module 202 is used to automatically generate virtual routing rules on the working node after adding the configured working node to the distributed container orchestration management cluster.

[0137] The virtual routing rule includes a mapping relationship between a virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster;

[0138] The changing module 203 is configured to change the configuration of the components of the working node by using the virtual routing rule, and restart the components of the working node.

[0139] In some embodiments, the distributed container orchestration management cluster to be deployed is a Kubernetes cluster, and the configuration module is specifically used to:

[0140] Configure the cluster server of the kubelet component and kube-proxy component of the worker node to the physical IP address of any of the Master nodes;

[0141] Set the forwarding mode of the kube-proxy component of the working node to ipvs mode and the scheduling algorithm to wrr algorithm.

[0142] In some embodiments, the virtual routing rule is a Linux virtual server LVS rule, and the virtual IP address in the LVS rule is the cluster IP of the kubernetes service of the Kubernetes cluster, wherein the cluster IP of the kubernetes service is the first address in the Cluster IP address pool.

[0143] Specifically, in this embodiment, the generation module 202 is specifically used to obtain the change information of the services and endpoints created by the Kubernetes cluster in real time through the kube-proxy component on the working node through the detection interface of the Kubernetes cluster, and automatically generate and maintain the LVS rules on the working node.

[0144] In some embodiments, the changing module 203 is specifically used to change the cluster server of the kubelet component and the kube-proxy component of the working node to the clusterIP of the kubernetes service in the LVS rule.

[0145] In some embodiments, as Figure 7As shown, the apparatus 200 further includes: an updating module 204 .

[0146] The update module 204 is used to delete the physical IP address of the failed Master node from the virtual routing rules through the Kubernetes cluster when a failure is detected in any of the Master nodes in the Kubernetes cluster, so as to update the virtual routing rules on all working nodes in the Kubernetes cluster.

[0147] For matters not described in detail in the apparatus for deploying a distributed container orchestration management cluster provided in the embodiments of the present application, reference may be made to the method for deploying a distributed container orchestration management cluster provided in the above embodiments. The apparatus for deploying a distributed container orchestration management cluster provided in the embodiments of the present application can achieve the same beneficial effects as those achieved by the method for deploying a distributed container orchestration management cluster provided in the above embodiments, and will not be further described here.

[0148] Based on the same inventive concept, the embodiment of the present application further provides an electronic device, the structural diagram of which is shown in FIG. Figure 8 As shown, the electronic device 300 includes at least one processor 301, a memory 302 and a bus 303, and the at least one processor 301 is electrically connected to the memory 302; the memory 302 is configured to store at least one computer-executable instruction, and the processor 301 is configured to execute the at least one computer-executable instruction, thereby performing any steps of the method for deploying a distributed container orchestration management cluster provided in any embodiment or any optional implementation method in Example 1 of the present application.

[0149] Furthermore, the processor 301 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Processing Unit (CPU).

[0150] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method for deploying a distributed container orchestration management cluster provided by the above method embodiment.

[0151] The computer-readable storage media provided in the embodiments of the present application include, but are not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, a readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0152] The present application also provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method for deploying a distributed container orchestration management cluster according to each of the above method embodiments.

[0153] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium.

[0154] Those skilled in the art will appreciate that each block in these structural diagrams and / or block diagrams and / or flow charts, as well as combinations of blocks in these structural diagrams and / or block diagrams and / or flow charts, can be implemented using computer program instructions. Those skilled in the art will appreciate that these computer program instructions can be provided to a general-purpose computer, a specialized computer, or a processor of other programmable data processing methods for implementation, thereby executing the schemes specified in the blocks or multiple blocks of the structural diagrams and / or block diagrams and / or flow charts disclosed in this application through the processor of the computer or other programmable data processing method.

[0155] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0156] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for deploying a distributed container orchestration management cluster, characterized in that: The distributed container orchestration management cluster to be deployed is a Kubernetes cluster, which includes multiple master nodes and multiple worker nodes. The method includes: Configuring a worker node based on the physical IP address of any of the multiple master nodes in the distributed container orchestration management cluster to be deployed, including: configuring the kubelet component of the worker node and the clusterserver of the kube-proxy component to the physical IP address of any of the multiple master nodes; setting the forwarding mode of the kube-proxy component of the worker node to ipvs mode and the scheduling algorithm to wrr algorithm; After adding the configured working node to the distributed container orchestration management cluster, a virtual routing rule is automatically generated on the working node, wherein the virtual routing rule includes a mapping relationship between a virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster. The virtual routing rule is a Linux virtual server LVS rule, and the virtual IP address in the LVS rule is the cluster IP of the kubernetesservice of the Kubernetes cluster. The cluster IP of the kubernetes service is the first address in the Cluster IP address pool. The LVS rule is used for the working node to access the multiple Master nodes. Change the cluster server of the kubelet component and kube-proxy component of the worker node to the cluster IP of the kubernetes service in the LVS rule, and restart the components of the worker node to make the api-server service on each Master node highly available for the worker node, so that the worker node uses the LVS rule as a reverse proxy to access each Master node.

2. The method according to claim 1, characterized in that Automatically generate virtual routing rules on the working node, including: The kube-proxy component on the working node obtains the change information of the services and endpoints created by the Kubernetes cluster in real time through the detection interface of the Kubernetes cluster, and automatically generates and maintains the LVS rules on the working node.

3. The method according to any one of claims 1 to 2, characterized in that Also includes: When a failure is detected in any of the Master nodes in the Kubernetes cluster, the physical IP address of the failed Master node is deleted from the virtual routing rules through the Kubernetes cluster to update the virtual routing rules on all working nodes in the Kubernetes cluster.

4. A device for deploying a distributed container orchestration management cluster, characterized in that: The distributed container orchestration management cluster to be deployed is a Kubernetes cluster, which includes multiple master nodes and multiple worker nodes. The device includes: A configuration module is used to configure the cluster server of the kubelet component and kube-proxy component of the working node to the physical IP address of any master node among the multiple master nodes; set the forwarding mode of the kube-proxy component of the working node to ipvs mode and the scheduling algorithm to wrr algorithm; A generation module is used to automatically generate virtual routing rules on the configured working node after adding the working node to the distributed container orchestration management cluster; The virtual routing rule includes a mapping relationship between a virtual IP address and the physical IP addresses of all Master nodes in the distributed container orchestration management cluster. The virtual routing rule is a Linux virtual server LVS rule. The virtual IP address in the LVS rule is the cluster IP of the kubernetes service of the Kubernetes cluster. The cluster IP of the kubernetes service is the first address in the cluster IP address pool. The LVS rule is used for the worker node to access the multiple Master nodes. The change module is used to change the cluster server of the kubelet component and kube-proxy component of the working node to the cluster IP of the kubernetes service in the LVS rule, and restart the components of the working node to make the api-server service on each Master node highly available for the working node, so that the working node uses the LVS rule as a reverse proxy to access each Master node.

5. The device according to claim 4, characterized in that The generation module is specifically used to obtain the change information of the services and endpoints created by the Kubernetes cluster in real time through the kube-proxy component on the working node through the detection interface of the Kubernetes cluster, and automatically generate and maintain the LVS rules on the working node.

6. The device according to any one of claims 4 to 5, characterized in that Also includes: An update module is used to delete the physical IP address of the failed Master node from the virtual routing rules through the Kubernetes cluster when a failure of any of the Master nodes in the Kubernetes cluster is detected, so as to update the virtual routing rules on all working nodes in the Kubernetes cluster.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for deploying a distributed container orchestration management cluster as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for deploying a distributed container orchestration management cluster as described in any one of claims 1 to 3.

Images