Multi-cluster remote control method and cluster resource exposure method

Abstract

The application discloses a multi-cluster remote control method and a cluster resource exposure method, and belongs to the technical field of cluster control. A P2P overlay network is established through a shared networking token token, and logical service identification (serviceUUID) is used for addressing. On the overlay network, the following is realized: (a) a control cluster declaratively issues a list to a managed cluster; (b) cross-cluster service tunneling and mirroring; and (c) label-based Sidecar injection type bidirectional exposure. The multi-cluster control is decoupled from the underlying network that specifically carries IP packets, and no additional underlying network configuration is required except for the shared token.

Description

Technical Field

[0001] This invention relates to the field of cluster control technology, specifically a method for remote control of multiple clusters and a method for exposing cluster resources. Background Technology

[0002] In existing scenarios involving interconnection, remote control, and service exposure of multiple Kubernetes clusters (or multiple distributed clusters), the over-reliance on fixed IPs, dedicated lines, VPNs, unified API gateways, and complex network pre-configurations leads to the following technical problems:

[0003] 1. Cross-cluster connectivity depends on the underlying network planning: Traditional multi-cluster management requires that each cluster's API Server and Agent have reachable private / public IP addresses, routes, firewall access, or VPN tunnels; in edge, home broadband NAT, multi-carrier, and cross-regional scenarios, the pre-configuration cost is high and frequent changes can easily lead to disconnection.

[0004] 2. The remote control plane and data plane are coupled by IP:Port: When the control terminal issues a resource manifest or executes operation and maintenance instructions to the managed cluster, it usually needs to point to a fixed endpoint of the target cluster. After cluster migration, node migration or IP change, the control relationship needs to be manually rebuilt.

[0005] 3. Cross-cluster service exposure methods are singular and vulnerable: Exposing services within cluster A to cluster B often relies on Ingress, LoadBalancer, NodePort, or dedicated line mirrors, making it difficult to establish bidirectional, declarative logical service channels under conditions of "no public IP address and no unified entry point".

[0006] 4. Separation of security boundaries and networking tokens: Some P2P solutions only address transmission reachability and fail to form an integrated management and control link with the cluster's native declarative API (CRD), RBAC, and shared networking tokens. This makes it difficult to meet the engineering requirements of "remote control with least privileges + zero-trust networking." Here, CRD is a custom resource. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to provide a multi-cluster remote control method and a cluster resource exposure method. By establishing a p2p overlay network through a shared networking token and addressing with logical service identifiers (serviceUUID), the following are implemented on the overlay network: (a) the control cluster declaratively issues a list to the managed cluster; (b) cross-cluster service tunnels and mirroring; and (c) tag-based sidecar injection bidirectional exposure, so that the multi-cluster control and service connectivity are decoupled from the underlying network that specifically carries IP packets, and no additional underlying network networking configuration is required except for the shared token.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] A multi-cluster remote control method includes the following steps:

[0010] (S100) Specifies the control cluster and the controlled cluster;

[0011] (S110) Deploy the first network credential module, the first p2p interconnect controller, and the remote control agent in the control cluster, and deploy the second network credential module, the second p2p interconnect controller, and the API agent Pod in the controlled cluster. libp2p is configured in both the first p2p interconnect controller and the second p2p interconnect controller; libp2p is a blockchain module.

[0012] (S120) Both the control cluster and the controlled cluster generate authentication tokens corresponding to each cluster through the command-line interface, and write the authentication tokens of the control cluster and the controlled cluster into a Secret file with the same name as the cluster in each cluster; where Secret is a key file.

[0013] (S130) Connect the control cluster and the controlled cluster to the same p2p network. The controlled cluster creates an account with the permission to operate and manage the resources of the controlled cluster and binds the account to the API proxy Pod in the controlled cluster.

[0014] (S140) The first p2p interconnect controller of the control cluster loads the Secret stored in the control cluster, and then issues manifests resources containing authentication tokens to the controlled cluster; where manifests are a list.

[0015] (S150) After the controlled cluster receives the manifests resource, the second p2p interconnect controller in the controlled cluster loads the Secret stored in the controlled cluster and compares the authentication token carried in the manifests resource with the authentication token stored in the controlled cluster. When the two authentication tokens are the same, the control cluster and the controlled cluster establish p2p network communication. Then, the control cluster sends an instruction to the API agent Pod of the controlled cluster, and the API agent Pod executes the instruction through the account created in step S130, thereby realizing the remote control of the controlled cluster by the control cluster.

[0016] In the above-mentioned multi-cluster remote control method, the manifest resources issued by the control cluster are generated in the control cluster, and the account and list are specified in the manifest resources.

[0017] In the above-described multi-cluster remote control method, the controlled cluster only grants necessary operations to the account.

[0018] In the above multi-cluster remote control method, when the controlling cluster deletes manifests resources, the controlled cluster corresponding to those manifests resources deletes the resource Pods corresponding to those manifests resources.

[0019] In the above multi-cluster remote control method, the control cluster's control over the controlled cluster is bound to the API agent Pod by the RBAC in the controlled cluster, where RBAC is the permission management function module.

[0020] In the aforementioned multi-cluster remote control method, when the control cluster sends manifests resources containing authentication tokens to the controlled cluster, the remote control agent in the control cluster resolves the p2p route next hop according to the mapping relationship between the authentication token and the controlled cluster.

[0021] A method for exposing cluster resources exposes the resources of a service-providing cluster to a service-consuming cluster belonging to the same P2P network. This P2P network is one that allows remote control of the clusters via the aforementioned multi-cluster remote control method. The specific operation is as follows:

[0022] (S210) Create a network Tunnel Custom Resource in the service provider cluster, wherein the TunnelCustom Resource is a tunnel-custom resource;

[0023] (S220) Create a network management response corresponding to the Tunnel Custom Resource in the service consumption cluster and generate a ClusterIP Service, wherein the ClusterIP Service is a cluster IP service;

[0024] (S230) Pods in the service consumption cluster access the service provider cluster through ClusterIP Service.

[0025] In the above cluster resource exposure method, step S210 involves mirroring the accessible Host service in the service providing cluster to the Service network of the service consuming cluster through the following steps:

[0026] (S211) Create a business Pod with a network tunnel tag in the service provider cluster;

[0027] (S212) The service provides that the p2p interconnection controller in the cluster detects the network tunnel label, injects the sidecar container into the business Pod created in step S211, and exposes the connection on the specified target_port, where sidecar is an auxiliary container and target_port is the target port.

[0028] In the above-described cluster resource exposure method, in step S220, the service consumer cluster establishes a reverse channel, enabling the service provider cluster and the service consumer cluster to communicate and connect through the network tunnel and the reverse channel.

[0029] The above-mentioned cluster resource exposure method allows the service consumer cluster to establish a reverse channel through a network tunnel or a proxy Pod with a network tunnel tag.

[0030] The technical solution of the present invention achieves the following beneficial technical effects:

[0031] 1. Due to the use of token+p2p networking, multiple clusters can be interconnected without VPN / dedicated lines, reducing network pre-configuration costs and time.

[0032] 2. Because serviceUUID+tag addressing is used, changes to the API Server Underlay address do not affect the p2p service registration logic, thus improving control continuity.

[0033] 3. Due to the declarative deployment of Manifests, the result is that remote operation and maintenance are unified with GitOps processes, improving the level of automation.

[0034] 4. Due to the MarkTunnel + Sidecar dual mode, the result is that it simultaneously covers the cross-cluster exposure of Host network segment services and existing Pods, expanding the applicable scenarios.

[0035] 5. Because RBAC binds to the proxy Pod, even if the P2P network shares the token, the specific scope of damage is still limited by Kubernetes permissions, thus improving security. Attached Figure Description

[0036] Figure 1 This is a flowchart of the multi-cluster remote control process in this invention;

[0037] Figure 2 This is a flowchart of the cluster resource exposure process in this invention. Detailed Implementation

[0038] The present invention will be further explained below with reference to examples.

[0039] The multi-cluster remote control method and cluster resource exposure method in this invention can be applied to existing Kubernetes cluster systems. Therefore, this embodiment will describe the technical solution involved in this invention by referring to its application in a Kubernetes cluster system.

[0040] like Figure 1 As shown, remote control of multiple clusters in a Kubernetes cluster system is achieved through the following method:

[0041] (S100) Specifies the control cluster and the controlled cluster.

[0042] When applying the multi-cluster remote control method of this invention to an existing Kubernetes cluster system, or when using this method to build a new Kubernetes cluster system, it is necessary to designate one cluster as the control cluster and the remaining clusters as the controlled clusters.

[0043] (S110) Deploy the first network credential module, the first p2p interconnect controller and the remote control agent in the control cluster, and deploy the second network credential module, the second p2p interconnect controller and the API agent Pod in the controlled cluster. libp2p is configured in both the first p2p interconnect controller and the second p2p interconnect controller.

[0044] After specifying the control cluster and the controlled cluster, the control cluster and the controlled cluster need to be modified accordingly so that the control cluster and the controlled cluster can meet the usage requirements and facilitate the application of the multi-cluster remote control method of this invention in the Kubernetes cluster system.

[0045] Specifically, the following components are deployed in the control cluster:

[0046] The first network credential module is used to store authentication tokens, such as Secret (secretRef).

[0047] The first p2p interconnect controller (mark) is used to establish overlay connections between clusters;

[0048] The remote control proxy (mark-proxy) is used to parse manifests resources and deliver the YAML manifest in the manifests resources to the remote kubectl proxy endpoint corresponding to the target serviceUUID;

[0049] The declarative tunneling module is used to tunnel a specified host:port (or hostNetwork) to the peer cluster service network (ClusterIP, etc.), where host:port is the host IP address and port, and hostNetwork is the host network field.

[0050] Deploy the following components in the controlled cluster: a second network credential module, a second p2p interconnect controller, an API proxy Pod (agent-proxy), and a declarative tunnel module. The second network credential module is the same as the first network credential module, the second p2p interconnect controller is the same as the first p2p interconnect controller, and the API proxy Pod is used to run kubectl proxy, bind to ServiceAccount, and register services on p2p using the tag declaration mark.mycos.io / *. Here, kubectl proxy is the proxy operation cluster binary, and ServiceAccount is the account.

[0051] (S120) Both the control cluster and the controlled cluster generate authentication tokens corresponding to each cluster through the command line interface, and write the authentication tokens of the control cluster and the controlled cluster into the Secret (mysecret) with the same name as the cluster in each cluster.

[0052] In this step, the control cluster uses its built-in Command Line Interface (CLI) to generate an authentication token corresponding to the control cluster, and the controlled clusters use the built-in CLI to generate their own authentication tokens. Each controlled cluster has its own unique authentication token. These authentication tokens are then collected together and written into a Secret with the same name as the cluster in each cluster for subsequent control and resource exposure.

[0053] (S130) Connect the control cluster and the controlled cluster to the same p2p network. The controlled cluster creates an account with the permission to operate and manage the resources of the controlled cluster and binds the account to the API proxy Pod in the controlled cluster.

[0054] In addition to communication between the control cluster and the controlled clusters via a peer-to-peer (P2P) network, communication between the controlled clusters is also based on a P2P network. In a P2P network, clusters can establish connections directly or through relay cooperation. The P2P components provide modular capabilities such as transmission, security, multiplexing, and NAT traversal, allowing participants to communicate at the overlay layer without pre-agreing on fixed IP addresses.

[0055] (S140) The first p2p interconnect controller of the control cluster loads the Secret stored in the control cluster and then issues manifests resources containing authentication tokens to the controlled cluster.

[0056] Control the creation of manifests resources in the cluster (API group example: mark-proxy.mycos.io / vlalphal):

[0057] spec.serviceUUID: Target remote service name (e.g., foo)

[0058] spec.secretRef: Network secret name

[0059] spec.manifests: One or more YAML manifest strings (e.g., for creating a Pod).

[0060] metadata.labels: mark.mycos.io / name / service / target_port consistent with the route of the first p2p interconnect controller.

[0061] (S150) After the controlled cluster receives the manifests resource, the second p2p interconnect controller in the controlled cluster loads the Secret stored in the controlled cluster and compares the authentication token carried in the manifests resource with the authentication token stored in the controlled cluster. When the two authentication tokens are the same, the control cluster and the controlled cluster establish p2p network communication. Then, the control cluster sends an instruction to the API agent Pod of the controlled cluster, and the API agent Pod executes the instruction through the account created in step S130, thereby realizing the remote control of the controlled cluster by the control cluster.

[0062] Meanwhile, when specific resources listed in the manifests change, the control cluster can also issue instructions to the API agent Pod in the controlled cluster, and the API agent Pod can execute the instructions through the account to perform operations such as creating, deleting, updating or adding resources of the resource Pod.

[0063] In this invention, the serviceUUID can be created by the API proxy Pod after the controlled cluster environment is prepared, or it can be created by the API proxy Pod after the controlled cluster is connected to the p2p network. Specifically, the controlled cluster announces its local proxy listening port to the p2p network through the tags mark.mycos.io / name (corresponding to the Secret name), mark.mycos.io / service (serviceUUID), mark.mycos.io / target_port, and mark.mycos.io / direction (e.g., mark), allowing peers (other clusters, including other controlled clusters of the control cluster) to connect.

[0064] When creating a serviceUUID, the controlled cluster restricts the account's operations, granting only necessary permissions. By combining the agent-proxy + ServiceAccount least-privilege remote API proxy structure with the p2p channel, a closed loop of "reachability + controllability + auditability" is formed. This ensures that even if the p2p network shares tokens, the specific scope of damage is still limited by Kubernetes permissions, thus improving security.

[0065] Furthermore, the control cluster and the controlled cluster in this invention can be located behind different translation addresses (NAT), with the first p2p interconnect controller and the second p2p interconnect controller responsible for NAT traversal / relay, eliminating the need for administrators to configure static routes or port mappings for both parties.

[0066] In the above multi-cluster remote control method, cross-cluster control and traffic do not use the API Server public IP as the addressing primary key, but instead use the p2p network selected by the token + serviceUUID + label port to determine the delivery target. This ensures that changes to the API Server Underlay address do not affect the p2p service registration logic, thus improving the continuity of control.

[0067] Moreover, when the control cluster and the controlled cluster start, the first p2p interconnect controller and the second p2p interconnect controller start. Then, libp2p (modular p2p network stack) determines whether the authentication token stored in the cluster is consistent with the token required for p2p network connection. If the authentication token is inconsistent, the connection is refused. This makes it possible for the Kubernetes cluster system managed by the multi-cluster remote control method in this invention to not require additional Underlay-specific configuration in addition to the regular Kubernetes components.

[0068] When using a control cluster to remotely control a controlled cluster, and when both the control cluster and the controlled cluster are connected to a p2p network, one cluster in this cluster system can provide services to another cluster. In this case, the resources in the service-providing cluster need to be exposed to the service-consuming cluster.

[0069] like Figure 2 As shown, the present invention also provides a method for exposing cluster resources, which specifically achieves the exposure of cluster resources through the following steps:

[0070] (S210) Create a networked Tunnel Custom Resource in the service provider cluster;

[0071] Specifically, as follows:

[0072] spec.serviceUUID: Overlay service name (e.g., foo2)

[0073] spec.secretRef: Secret name

[0074] spec.host / spec.port: e.g., 192.168.1.1:80

[0075] spec.hostNetwork: true: indicates that the network is reachable via the node.

[0076] In this step, the user creates a service Pod (without a sidecar) with the markTunnel tag in the service provider cluster. When the p2p interconnect controller in the service provider cluster detects the appearance of the markTunnel tag, the service provider cluster injects a sidecar container into the service Pod, exposing the connection on the specified target_port. Using the markTunnel + sidecar dual-mode, cross-cluster exposure of both host network segment services and existing Pods can be covered simultaneously, expanding the applicable scenarios.

[0077] (S220) Create a network management response corresponding to the Tunnel Custom Resource in the service consumption cluster and generate a ClusterIP Service;

[0078] Specifically, as follows:

[0079] spec.inbound: true

[0080] spec.host / port: Local loopback proxy port (e.g., 127.0.0.1:8080)

[0081] spec.serviceSpec: Generates a ClusterIP Service (port 8080) in the service consumer cluster.

[0082] In this step, after the service consumer cluster creates a network management response, it will establish a reverse channel through its own markTunnel or another Deployment with direction=mark (such as a local kubectl proxy), thereby realizing bidirectional exposure between the service provider cluster and the service consumer cluster.

[0083] (S230) Pods in the service consumption cluster access the service provider cluster through ClusterIP Service.

[0084] After generating a ClusterIP Service in the service consumption cluster, Pods within the service consumption cluster can access the service provider cluster's 192.168.1.1:80 via the ClusterIP, without the service provider cluster exposing a public network load balancer.

[0085] In the above cluster resource exposure method, sidecar injection only requires creating a Pod with the `markTunnel` tag. The `mark` tag in the service provider cluster will automatically inject the sidecar into this Pod. The sidecar connects to P2P on the `target_port` and registers on the P2P network using a service (`serviceUUID`). The service consumer cluster consumes the service through resources of type `markTunnel` with inbound attributes or proxy Pods with the same resource tag. The resource tag is used to declare the resource. This cluster resource exposure method is suitable for quickly connecting existing workloads to the P2P network.

[0086] By exposing cluster resources using the methods described above, existing Pods can be incorporated into the p2p service network without modifying the business image, reducing the cost of modifying existing Kubernetes cluster systems.

[0087] In addition, the present invention provides another method for exposing cluster resources. Specifically, a container (not a sidecar container) is injected into the control cluster, and a bidirectional link is exposed in this container. Then, a Pod is created, and the service provider cluster tunnel is transmitted to the service consumer cluster using network management response.

[0088] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of the claims of this patent application.

Claims

1. A multi-cluster remote control method, characterized in that, Includes the following steps: (S100) Specifies the control cluster and the controlled cluster; (S110) Deploy the first network credential module, the first p2p interconnect controller and the remote control agent in the control cluster, and at the same time deploy the second network credential module, the second p2p interconnect controller and the API agent Pod in the controlled cluster. libp2p is configured in both the first p2p interconnect controller and the second p2p interconnect controller. (S120) Both the control cluster and the controlled cluster generate authentication tokens corresponding to each cluster through the command line interface, and write the authentication tokens of the control cluster and the controlled cluster into the Secret with the same name as the cluster in each cluster. (S130) Connect the control cluster and the controlled cluster to the same p2p network. The controlled cluster creates an account with the permission to operate and manage the resources of the controlled cluster and binds the account to the API proxy Pod in the controlled cluster. (S140) The first p2p interconnect controller of the control cluster loads the Secret stored in the control cluster, and then sends manifests resources containing authentication tokens to the controlled cluster; (S150) After the controlled cluster receives the manifests resource, the second p2p interconnect controller in the controlled cluster loads the Secret stored in the controlled cluster and compares the authentication token carried in the manifests resource with the authentication token stored in the controlled cluster. When the two authentication tokens are the same, the control cluster and the controlled cluster establish p2p network communication. Then, the control cluster sends an instruction to the API agent Pod of the controlled cluster, and the API agent Pod executes the instruction through the account created in step S130, thereby realizing the remote control of the controlled cluster by the control cluster.

2. The multi-cluster remote control method according to claim 1, characterized in that, The manifests resources issued by the control cluster are generated in the control cluster, and the account and manifest are specified in the manifests resources.

3. The multi-cluster remote control method according to claim 1, characterized in that, A controlled cluster grants accounts only the necessary operations.

4. The multi-cluster remote control method according to claim 1, characterized in that, When the controlling cluster deletes a manifest resource, the controlled cluster corresponding to that manifest resource deletes the corresponding resource Pod.

5. The multi-cluster remote control method according to claim 1, characterized in that, The control cluster's control over the controlled cluster is achieved by the RBAC in the controlled cluster being bound to the API proxy Pod.

6. The multi-cluster remote control method according to claim 1, characterized in that, When the control cluster distributes manifest resources containing authentication tokens to the controlled clusters, the remote control agent in the control cluster resolves the next hop of the p2p route based on the mapping relationship between the authentication token and the controlled clusters.

7. A method for exposing cluster resources, characterized in that, The resources of the service provider cluster are exposed to a service consumer cluster belonging to the same p2p network, which is a network that remotely controls the cluster using the multi-cluster remote control method described in claim 1. Specifically, the operation is as follows: (S210) Create a networked Tunnel Custom Resource in the service provider cluster; (S220) Create a network management response corresponding to the Tunnel Custom Resource in the service consumption cluster and generate a ClusterIP Service; (S230) Pods in the service consumption cluster access the service provider cluster through ClusterIP Service.

8. The cluster resource exposure method according to claim 7, characterized in that, In step S210, the Host service accessible in the service provider cluster is mirrored to the Service network of the service consumer cluster through the following steps: (S211) Create a business Pod with a network tunnel tag in the service provider cluster; (S212) The service provides that the p2p interconnect controller in the cluster detects the network tunnel label, injects the sidecar container into the business Pod created in step S211, and exposes the connection on the specified target_port.

9. The cluster resource exposure method according to claim 8, characterized in that, In step S220, the service consumption cluster establishes a reverse channel, enabling the service provider cluster and the service consumer cluster to communicate and connect through the network tunnel and the reverse channel.

10. The cluster resource exposure method according to claim 9, characterized in that, The service consumption cluster establishes a reverse channel through a network tunnel or a proxy Pod labeled with a network tunnel.

Images