Cross-network multi-cloud multi-cluster application distribution method, device, equipment and medium

Abstract

The invention relates to the technical field of Kubernetes cloud platforms, and discloses a cross-network multi-cloud multi-cluster application distribution method, device and equipment and a medium, and the method comprises the steps: creating a server application and a k8s APIService resource in a central cluster; proxy applications are created in the multiple access clusters, and remote procedure call requests are initiated to an API server through the proxy applications; when the API server receives a remote procedure call request of the proxy application, creating a bidirectional long connection tunnel of the server application and the proxy application; and creating an access cluster key for application distribution of the continuous delivery tool in the central cluster, and distributing applications to the plurality of access clusters through the bidirectional long connection tunnel by adopting the continuous delivery tool based on the access cluster key. According to the method, the advanced characteristic of ArgoCD multi-cluster distribution is fully played, the user cost is reduced, and the risk of leakage of the configuration file of the user access cluster is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of Kubernetes cloud platform, and in particular to a method, device, equipment and medium for distributing multi-cloud and multi-cluster applications across networks. Background Art

[0002] With the widespread use of Kubernetes clusters (hereinafter referred to as k8s clusters), multi-cloud and multi-cluster has become a trend. Mainstream cloud vendors have launched distributed container cloud products. The cloud vendor provides a k8s cluster as the central cluster, and manages multiple k8s clusters in the cloud vendor, other cloud vendors and the user's local IDC (Internet Data Center) as access clusters. By deploying ArgoCD (Argo Continuous Delivery, CD) in the central cluster, applications are distributed and deployed to multiple access clusters and maintained in a unified manner.

[0003] ArgoCD is a continuous delivery tool based on GitOps. It uses GitOps to distribute applications to multiple k8s clusters. Its architecture is as follows: Figure 1 As shown. The ArgoCD application itself is deployed in the k8s cluster, and the kubeconfig (kubeconfig is a configuration file used by the Kubernetes client to interact with the k8s cluster) of the cluster to be deployed is stored in the form of a secret (key), and the secret name is used as the name of the cluster to be deployed. By creating an Application custom resource object, the Git / Helm (Helm plug-in) warehouse address of the application source, the name of the cluster to be deployed, and other information are configured in the resource object spec (spec is the spec attribute field in the k8s resource object, representing the expected value of the resource). ArgoCD automatically pulls the application yaml (a highly readable format used to express data serialization) configuration file in the Git / Helm warehouse, connects the rendered k8s native yaml with the kubeconfig saved in the secret, and pushes it to the cluster to be deployed. At the same time, it monitors the changes of the Git / Helm warehouse files and the application-related resources in the cluster to be deployed to ensure that the application resources in the cluster to be deployed are dynamically consistent with the expected values ​​in the warehouse.

[0004] It can be seen that ArgoCD's push mechanism strongly relies on the network accessibility from ArgoCD to the cluster to be deployed. In the distributed container cloud scenario, ArgoCD is deployed on the cloud vendor's central cluster, and the clusters to be deployed (i.e., access clusters) are scattered in the cloud vendor, other cloud vendors, and the user's local IDC. For security and cost considerations, the kube-apiserver service of the access cluster (a core component in the k8s cluster, i.e., the API server, which is mainly responsible for providing the entrance to the kube-apiserver) generally does not provide a public IP or domain name for access outside the cluster. As a result, ArgoCD on the cloud cannot directly access the k8s clusters in other VPCs of the cloud vendor, other cloud vendors, and the user's local IDC, which limits ArgoCD's multi-cloud and multi-cluster application distribution in cross-network scenarios.

[0005] In summary, when using ArgoCD to distribute applications to multiple Kubernetes access clusters, you need to provide the kubeconfig of the access cluster and the network between ArgoCD and the kube-apiserver of the access cluster is reachable. When the access cluster is in other VPCs or other cloud vendors or user-built IDCs, ArgoCD cannot be used for application distribution due to network unreachability. Summary of the invention

[0006] In view of this, the present invention provides a cross-network multi-cloud multi-cluster application distribution method, apparatus, device and medium to solve the problem of limitations when using continuous delivery tools to distribute applications to multi-access clusters in cross-network scenarios.

[0007] In a first aspect, the present invention provides a method for distributing multi-cloud and multi-cluster applications across networks, the method comprising:

[0008] Create a server application and k8s APIService resources in the pre-established central cluster, and listen for remote procedure call requests after the server application is started; the central cluster includes the API server;

[0009] Create proxy applications in multiple pre-established access clusters, and initiate remote procedure call requests to the API server through the proxy applications;

[0010] When the API server receives the remote procedure call request from the proxy application, it forwards it to the server application through the k8s APIService resource to create a bidirectional long connection tunnel between the server application and the proxy application;

[0011] Deploy the continuous delivery tool in the central cluster and create an access cluster key for the continuous delivery tool to distribute applications. Based on the access cluster key, use the continuous delivery tool to distribute applications to multiple access clusters through a bidirectional long connection tunnel.

[0012] The cross-network multi-cloud multi-cluster application distribution method provided by the present invention is a method for supporting continuous delivery tools (such as ArgoCD) to distribute applications to multiple k8s clusters across networks based on k8s APIService resources and reverse proxy tunnels. By deploying a server application and an agent application (hereinafter referred to as an agent application) in a central cluster and an access cluster respectively, the agent application actively requests an APIService address exposed by an API server (hereinafter referred to as kube-apiserver) of the central cluster, accesses the server application at the backend of the APIService, and the server application and the agent application establish a bidirectional long connection tunnel, which is forwarded through the long connection tunnel created by the server application and the agent application, and accesses the kube-apiserver of the access cluster to perform application distribution and resource monitoring, thereby realizing that only one set of continuous delivery tools needs to be deployed in the central cluster, and there is no need to deploy continuous delivery tools in each access cluster, thereby increasing the resource utilization rate of customer clusters; at the same time, the continuous delivery tool can distribute applications to all access clusters, and can give full play to the advanced features of the continuous delivery tool based on cluster labels, differentiated and flexible deployment of multiple clusters, and solves the problem of being restricted when using continuous delivery tools to distribute applications to multiple access clusters in cross-network scenarios.

[0013] In an optional implementation manner, the central cluster and multiple access clusters are established in the following manner:

[0014] Create multiple k8s clusters, use one of the k8s clusters as the central cluster, and use the remaining k8s clusters as multiple access clusters; the central cluster includes the APIService address, CA certificate, client certificate and client key exposed as a public IP or domain name, and the APIService address exposed as a public IP or domain name is integrated on the API server.

[0015] The cross-network multi-cloud multi-cluster application distribution method provided by the present invention creates multiple k8s clusters, takes one of the k8s clusters as the central cluster, and uses the remaining k8s clusters except the central cluster as multiple access clusters; the central cluster includes an APIService address, a CA certificate, a client certificate, and a client key exposed in the form of a public IP or domain name, and the APIService address exposed in the form of a public IP or domain name is integrated on the API server, which is conducive to the proxy application accessing the APIService address on the API server.

[0016] In an optional implementation, creating a server application and a k8sAPIService resource in a pre-established central cluster includes:

[0017] Create a k8s APIService resource in the pre-established central cluster and point the k8s APIService resource to the server application so that you can access the server application by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server.

[0018] The cross-network multi-cloud and multi-cluster application distribution method provided by the present invention creates a k8s APIService resource in a pre-established central cluster, and points the k8s APIService resource to the server application, so as to access the server application by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server. It only needs to add a public IP or domain name to the kube-apiserver of the central cluster, and there is no need to configure a public IP or domain name in each access cluster, thereby reducing user costs, increasing user willingness to access a distributed container cloud, and providing conditions for establishing a two-way long connection tunnel between the server application and the proxy application.

[0019] In an optional implementation, creating a proxy application in each of the pre-established multiple access clusters, and initiating a remote procedure call request to the API server through the proxy application includes:

[0020] Create proxy applications containing startup parameters in multiple pre-established access clusters. The startup parameters include the APIService address of the central cluster exposed in the form of public IP or domain name.

[0021] When the proxy application is started, a remote procedure call request is initiated to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters.

[0022] The cross-network multi-cloud and multi-cluster application distribution method provided by the present invention creates a proxy application containing startup parameters in multiple pre-established access clusters, and the startup parameters include the APIService address of the central cluster exposed in the form of a public IP or domain name. When the proxy application is started, a remote procedure call request is initiated to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters, thereby achieving the purpose of accessing the server application.

[0023] In an optional implementation, when the API server receives the remote procedure call request of the proxy application, it forwards it to the server application through the k8s APIService resource to create a bidirectional long connection tunnel between the server application and the proxy application, including:

[0024] When the API server receives the remote procedure call request from the proxy application, it forwards the remote procedure call request to the server application according to the URL path matching k8sAPIService resource. When the server application receives the remote procedure call request, the server application communicates with the proxy application and creates a bidirectional long connection tunnel between the server application and the proxy application.

[0025] The cross-network multi-cloud and multi-cluster application distribution method provided by the present invention, when the API server receives the remote procedure call request of the proxy application, it forwards the remote procedure call request to the server application according to the URL path matching k8s APIService resource, when the server application receives the remote procedure call request, it realizes the communication between the server application and the proxy application, and creates a two-way long connection tunnel between the server application and the proxy application, which provides conditions for subsequent application distribution.

[0026] In an optional implementation, deploying a continuous delivery tool in a central cluster and creating an access cluster key for use in continuous delivery tool application distribution includes:

[0027] Assemble the APIService address of the central cluster exposed in the form of public IP or domain name into the API server address of the access cluster, use the CA certificate of the central cluster as the CA certificate of the access cluster, use the client certificate of the central cluster as the client certificate of the access cluster, and use the client key of the central cluster as the client key of the access cluster to create the access cluster key for the continuous delivery tool application distribution.

[0028] The cross-network multi-cloud multi-cluster application distribution method provided by the present invention enables ArgoCD located in the central cluster to use the certificate private key of the central cluster to request the API server address of the central cluster, and forward it through the tunnel created by the server application and the agent application to access the API server (kube-apiserver) of the access cluster for application distribution and resource monitoring, so as to give full play to the advanced features of ArgoCD multi-cluster distribution, reduce user costs, and reduce the risk of leakage of kubeconfig configuration files when users access the cluster.

[0029] In an optional implementation, distributing applications to multiple access clusters through a bidirectional persistent connection tunnel using a continuous delivery tool based on an access cluster key includes:

[0030] When the continuous delivery tool distributes applications to multiple access clusters, the name of the access cluster key is used as the name of the distributed access cluster.

[0031] The cross-network multi-cloud multi-cluster application distribution method provided by the present invention uses the name of the access cluster key as the name of the distributed access cluster when the continuous delivery tool distributes applications to multiple access clusters. The access clusters are distinguished by the name, which facilitates the management and resource monitoring of multiple access clusters.

[0032] In a second aspect, the present invention provides a cross-network multi-cloud multi-cluster application distribution device, the device comprising:

[0033] The central cluster configuration module is used to create server applications and k8sAPIService resources in the pre-established central cluster, and listen to remote procedure call requests after the server application is started;

[0034] An access cluster configuration module is used to create proxy applications in multiple pre-established access clusters, and initiate a remote procedure call request to the API server through the proxy application;

[0035] A bidirectional persistent connection tunnel creation module is used to forward the remote procedure call request from the proxy application to the server application through the k8s APIService resource after the API server receives it, so as to create a bidirectional persistent connection tunnel between the server application and the proxy application;

[0036] The application distribution module is used to deploy continuous delivery tools in the central cluster and create access cluster keys for continuous delivery tool application distribution. Based on the access cluster keys, the continuous delivery tools are used to distribute applications to multiple access clusters through bidirectional long connection tunnels.

[0037] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the cross-network multi-cloud and multi-cluster application distribution method of the first aspect or any corresponding embodiment thereof.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the cross-network multi-cloud and multi-cluster application distribution method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0039] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the cross-network multi-cloud and multi-cluster application distribution method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is the existing ArgoCD architecture diagram;

[0042] Figure 2 is a flow chart of a method for distributing multi-cloud and multi-cluster applications across networks according to an embodiment of the present invention;

[0043] Figure 3 is a flow chart of another cross-network multi-cloud multi-cluster application distribution method according to an embodiment of the present invention;

[0044] Figure 4 is a flowchart of another cross-network multi-cloud multi-cluster application distribution method according to an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of the architecture of multi-cloud and multi-cluster application distribution across networks according to an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of another architecture of multi-cloud and multi-cluster application distribution across networks according to an embodiment of the present invention;

[0047] Figure 7 It is a schematic diagram of the relationship between kube-apiserver, k8s APIService resources and server applications according to an embodiment of the present invention;

[0048] Figure 8 is a structural block diagram of a cross-network multi-cloud multi-cluster application distribution device according to an embodiment of the present invention;

[0049] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] According to an embodiment of the present invention, an embodiment of a method for distributing multi-cloud and multi-cluster applications across networks is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0052] In this embodiment, a cross-network multi-cloud multi-cluster application distribution method is provided, which can be used for the above-mentioned computer terminals, such as central processing units, servers, etc. Figure 2 is a flowchart of a multi-cloud and multi-cluster application distribution method across networks according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0053] Step S101, create a server application and k8s APIService resources in a pre-established central cluster, and listen to remote procedure call requests after the server application is started; the central cluster includes an API server.

[0054] Specifically, k8s is the abbreviation of Kubernetes, which is an open source container cluster management system used to automatically deploy, expand and manage containerized applications.

[0055] The server application is also called the service-side application. The server application is deployed in the central cluster as a workload. The API server is the k8s apiserver. The k8s APIServce is a resource object deployed in the central cluster along with the server application. Kubernetes allows users to register custom API services in the cluster and access the API services through the Kubernetes API server. When a user creates a k8s APIService, Kubernetes creates a RESTful interface for the API service and generates a uniform resource name (URN) for it. Through the uniform resource name URN, other components and users can use the API server k8s apiserver to access and operate the API service. In the central cluster, the k8s APIService resource serves as the front-end application, and the server application serves as the back-end application. The relationship between kube-apiserver, k8s APIService, and server applications is as follows: Figure 7 shown.

[0056] A remote procedure call request is referred to as a gRPC (gRemote Procedure Call) request. It is a remote procedure call request. After the server application is started, it listens to the gRPC request of the access cluster in real time.

[0057] Step S102: Create proxy applications in multiple pre-established access clusters, and initiate a remote procedure call request to the API server through the proxy application.

[0058] Specifically, Figure 6 In step 1 shown in the figure, the proxy application is also called the agent application. The agent application initiates a remote procedure call request to the kube-apiserver through the APIService address that carries the public IP or domain name of the central cluster in the startup parameters. The APIService address is the abbreviation of the k8s APIService address.

[0059] Step S103, when the API server receives the remote procedure call request of the proxy application, it forwards it to the server application through the k8sAPIService resource to create a bidirectional long connection tunnel between the server application and the proxy application.

[0060] Specifically, Figure 6Steps 2, 3, and 4 are shown. After receiving the request, the kube-apiserver of the central cluster matches the k8s APIService resource based on the URL path, and forwards the request to the server application according to the backend service bound to the k8s APIServer. The server application and the agent application establish a bidirectional long connection tunnel, also known as a gRPC long connection tunnel.

[0061] Step S104, deploying a continuous delivery tool in the central cluster, and creating an access cluster key for the continuous delivery tool to distribute applications, and distributing applications to multiple access clusters through a bidirectional long connection tunnel using the continuous delivery tool based on the access cluster key.

[0062] Specifically, the continuous delivery tool refers to ArgoCD. ArgoCD is a continuous delivery tool based on the GitOps concept (GitOps is a model for achieving continuous delivery. Its core idea is to store the declarative infrastructure and applications of the application system in the Git version control library). The configuration repository (such as Git / Helm repository, etc.) is the only trusted source for application deployment and management. Users define the expected state of the application in the configuration repository. ArgoCD continuously monitors and pulls the configuration in the repository, delivers the application to the specified k8s cluster according to the expected state, and continuously monitors the changes of resources in the k8s cluster to ensure that the actual resources in the k8s cluster are consistent with the expected values ​​in the repository. The access cluster key is also called the access cluster secret.

[0063] Exemplarily, when distributing applications, ArgoCD uses the APIService address that carries the public IP or domain name of the central cluster kube-apiserver, the CA certificate of the central cluster, the client certificate of the central cluster, and the client key of the central cluster to create the ArgoCD access cluster secret for use in ArgoCD application distribution and resource monitoring.

[0064] The cross-network multi-cloud multi-cluster application distribution method provided in this embodiment is a method for supporting continuous delivery tools (such as ArgoCD) to distribute applications to multiple k8s clusters across networks based on k8s APIService resources and reverse proxy tunnels. By deploying a server application and an agent application (hereinafter referred to as an agent application) in a central cluster and an access cluster respectively, the agent application actively requests the APIService address exposed by the API server (hereinafter referred to as kube-apiserver) of the central cluster to access the server application at the backend of the APIService. The server application and the agent application establish a bidirectional long connection tunnel, which is forwarded through the long connection tunnel created by the server application and the agent application to access the kube-apiserver of the access cluster for application distribution and resource monitoring. This achieves the goal of deploying only one set of continuous delivery tools in the central cluster, without deploying continuous delivery tools in each access cluster, thereby increasing the resource utilization of customer clusters. At the same time, the continuous delivery tool can distribute applications to all access clusters, which can give full play to the advanced features of the continuous delivery tool based on cluster labels, differentiated and flexible deployment of multiple clusters, and solves the problem of being restricted when using continuous delivery tools to distribute applications to multiple access clusters in cross-network scenarios.

[0065] In this embodiment, a cross-network multi-cloud multi-cluster application distribution method is provided, which can be used for the above-mentioned computer terminals, such as central processing units, servers, etc. Figure 3 is a flowchart of a multi-cloud and multi-cluster application distribution method across networks according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0066] Step S201, create a server application and k8s APIService resources in a pre-established central cluster, and listen to remote procedure call requests after the server application is started; the central cluster includes an API server.

[0067] Specifically, the above step S201 includes:

[0068] Step a. Create a k8s APIService resource in the pre-established central cluster and point the k8s APIService resource to the server application so that the server application can be accessed by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server.

[0069] Among them, Figure 5As shown, the central cluster and multiple access clusters are established in the following way: create multiple k8s clusters, use one of the k8s clusters as the central cluster, and use the remaining k8s clusters except the central cluster as multiple access clusters; the central cluster includes the APIService address, CA certificate, client certificate and client key exposed in the form of public IP or domain name, and the APIService address exposed in the form of public IP or domain name is integrated on the API server, which is assembled from the public IP or domain name and the URN corresponding to the k8s APIService.

[0070] like Figure 6 As shown, the deployment, configmap (an API object used to store configuration data), secret (key), service (service) and corresponding rbac (Role-Based Access Control) resource request of the server application are created in the central cluster.

[0071] Create a k8s APIService resource in the central cluster and point the k8s APIService resource to the backend server application. The server application can be accessed by accessing the APIService address composed of the public IP or domain name bound to the API server and the URN corresponding to the k8s APIService.

[0072] Step S202: Create proxy applications in multiple pre-established access clusters, and initiate a remote procedure call request to the API server through the proxy application.

[0073] Specifically, the above step S202 includes:

[0074] Step S2021: Create a proxy application including startup parameters in each of the pre-established multiple access clusters, where the startup parameters include the APIService address of the central cluster exposed in the form of a public IP or domain name.

[0075] Specifically, the agent application is referred to as an agent application, which refers to an autonomous and intelligent system that can perceive the external environment, make decisions and execute corresponding actions to achieve a predetermined goal. In this embodiment, the agent application is deployed in each access cluster and establishes a communication connection with the server application of the central cluster.

[0076] Step S2022: When the proxy application is started, a remote procedure call request is initiated to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters.

[0077] Specifically, when the agent application is started, the agent application initiates a request to the kube-apiserver of the central cluster through the APIService address that carries the public IP or domain name of the central cluster in the startup parameters.

[0078] Step S203: When the API server receives the remote procedure call request from the proxy application, it forwards it to the server application through the k8sAPIService resource to create a bidirectional long connection tunnel between the server application and the proxy application. Figure 2 Step S103 of the illustrated embodiment will not be described in detail here.

[0079] Step S204: deploy the continuous delivery tool in the central cluster, create an access cluster key for the continuous delivery tool to distribute applications, and use the continuous delivery tool to distribute applications to multiple access clusters through a bidirectional persistent connection tunnel based on the access cluster key. Figure 2 Step S104 of the illustrated embodiment will not be described in detail here.

[0080] The cross-network multi-cloud and multi-cluster application distribution method provided in this embodiment creates a k8s APIService resource in a pre-established central cluster, and points the k8s APIService resource to the server application, so as to access the server application by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server. It only needs to add a public IP or domain name to the API server of the central cluster (referred to as kube-apiserver), and there is no need to configure a public IP or domain name in each access cluster, thereby reducing user costs, increasing user willingness to access the distributed container cloud, and providing conditions for the API server application and the proxy application to establish a two-way long connection tunnel. A proxy application containing startup parameters is created in each of the pre-established multiple access clusters, and the startup parameters include the APIService address of the central cluster exposed in the form of a public IP or domain name, so that when the proxy application is started, a remote procedure call request is initiated to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters, thereby achieving the purpose of accessing the server application.

[0081] In this embodiment, a cross-network multi-cloud multi-cluster application distribution method is provided, which can be used for the above-mentioned computer terminals, such as central processing units, servers, etc. Figure 4 is a flowchart of a multi-cloud and multi-cluster application distribution method across networks according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0082] Step S301: Create a server application and k8s APIService resources in the pre-established central cluster, and listen for remote procedure call requests after the server application is started; the central cluster includes an API server. Figure 3 Step S201 of the illustrated embodiment will not be described in detail here.

[0083] Step S302: Create proxy applications in multiple pre-established access clusters, and initiate a remote procedure call request to the API server through the proxy application. Figure 3 Step S202 of the illustrated embodiment will not be described in detail here.

[0084] Step S303, when the API server receives the remote procedure call request of the proxy application, it forwards it to the server application through the k8sAPIService resource to create a bidirectional long connection tunnel between the server application and the proxy application.

[0085] Specifically, the above step S303 includes:

[0086] Step b. When the API server receives the remote procedure call request from the proxy application, it forwards the remote procedure call request to the server application based on the URL path matching k8s APIService resource. When the server application receives the remote procedure call request, the server application communicates with the proxy application and creates a bidirectional long connection tunnel between the server application and the proxy application.

[0087] Specifically, Figure 6 As shown in the figure, the URL path refers to the part of the network address used to specify the specific location of the resource. The central cluster kube-apiserver receives the remote procedure call request, matches the URN of k8s APIService according to the URL path, and forwards it to the server application. After the two communicate, a bidirectional long connection tunnel is established between the server application and the agent application, that is, the gRPC long connection tunnel.

[0088] Step S304: deploy a continuous delivery tool in the central cluster, create an access cluster key for the continuous delivery tool to distribute applications, and distribute applications to multiple access clusters using the continuous delivery tool through a bidirectional long connection tunnel based on the access cluster key.

[0089] Specifically, the above step S304 includes:

[0090] Step S3041, assemble the APIService address of the central cluster exposed in the form of public IP or domain name into the API server address of the access cluster, use the CA certificate of the central cluster as the CA certificate of the access cluster, use the client certificate of the central cluster as the client certificate of the access cluster, and use the client key of the central cluster as the client key of the access cluster to create an access cluster key for use in continuous delivery tool application distribution.

[0091] The access cluster key is also called the access cluster secret. It uses the APIService address exposed by the central cluster kube-apiserver with a public IP or domain name to assemble the API server address of the access cluster, the CA certificate of the central cluster, the client certificate of the central cluster, and the client key of the central cluster to create the continuous delivery tool ArgoCD access cluster secret for application distribution and resource monitoring by the continuous delivery tool ArgoCD.

[0092] Step S3042: When the continuous delivery tool distributes applications to multiple access clusters, the name of the access cluster key is used as the name of the distributed access cluster.

[0093] Specifically, when the continuous delivery tool ArgoCD distributes applications and monitors resources to the access cluster, the name of the access cluster secret created above is used as the name of the access cluster.

[0094] The cross-network multi-cloud multi-cluster application distribution method provided in this embodiment, when the API server receives the remote procedure call request of the proxy application, it matches the k8s APIService resource according to the URL path and forwards the remote procedure call request to the server application. When the server application receives the remote procedure call request, it realizes the communication between the server application and the proxy application, and creates a two-way long connection tunnel between the server application and the proxy application, which provides conditions for subsequent application distribution, and realizes that ArgoCD located in the central cluster uses the certificate private key of this central cluster to request the API server k8s APIService address of this central cluster, and forwards through the tunnel created by the server application and the agent application to access the API server k8s cluster resource object (kube-apiserver) of the access cluster for application distribution and resource monitoring, giving full play to the advanced features of ArgoCD multi-cluster distribution, reducing user costs, and reducing the risk of leakage of user access cluster kubeconfig configuration files. When the continuous delivery tool distributes applications to multiple access clusters, the name of the access cluster key is used as the name of the distributed access cluster, and the access clusters are distinguished by the name, which facilitates the management and resource monitoring of multiple access clusters.

[0095] As one or more specific application embodiments of the present invention, Figure 5 and Figure 6 The cross-network multi-cloud multi-cluster application distribution method provided by the present invention is further described in detail. Figure 6 As shown, the specific process is as follows:

[0096] Step 1. Figure 5 As shown in the figure, multiple Kubernetes clusters, namely k8s clusters, are created, one of which is planned as the central cluster and the others are planned as access clusters.

[0097] Step 2. Install and deploy the server application and k8s APIService resources (k8s cluster resource objects) in the central cluster, and mount the public IP or domain name.

[0098] Step 3. After the server application starts, listen for gRPC requests.

[0099] Step 4. Deploy the agent application in the access cluster.

[0100] Step 5. After the agent application is started, it uses the APIService address with the public IP or domain name of the central cluster in the startup parameters to initiate a request to the kube-apiserver of the central cluster.

[0101] Step 6. After the central cluster kube-apiserver receives the request, it forwards it to the backend server application for processing according to the k8s APIService resource configuration.

[0102] Step 7. After the server application receives the request from the agent application, it creates a bidirectional gRPC persistent connection tunnel.

[0103] Step 8. Create an access cluster secret in the central cluster for ArgoCD application distribution and resource monitoring, and configure the access cluster configuration file kubeconfig in the access cluster secret, that is, use the API server address of the central cluster and the URN corresponding to the k8s APIService to assemble the API server address of the access cluster, use the CA certificate of the central cluster as the CA certificate of the access cluster, use the client certificate of the central cluster as the client certificate of the access cluster, and use the client private key of the central cluster as the client private key of the access cluster, so that ArgoCD can perform application distribution and resource monitoring based on the access cluster secret.

[0104] Step 9. When ArgoCD distributes applications and resource monitoring to the access cluster, use the name of the access cluster secret created above as the name of the access cluster.

[0105] The cross-network multi-cloud multi-cluster application distribution method provided in this embodiment deploys server applications and agent applications in the central cluster and access cluster respectively, and the agent application actively requests the APIService address exposed by the central cluster kube-apiserver to access the server application at the backend of the APIService. The two establish a bidirectional long connection tunnel. The ArgoCD located in the central cluster uses the certificate private key of the central cluster to request the k8sAPIService address of the central cluster, and forwards through the tunnel created by the server and the agent to access the kube-apiserver of the access cluster to perform application distribution and resource monitoring. In this embodiment, only one set of ArgoCD needs to be deployed in the central cluster, and there is no need to deploy ArgoCD in each access cluster, which increases the resource utilization of the customer cluster; at the same time, ArgoCD can distribute applications to all access clusters, and can give full play to ArgoCD's advanced features such as cluster label-based, multi-cluster differentiated and flexible deployment. It is only necessary to add a public IP or domain name to the central cluster kube-apiserver, and there is no need to configure a public IP or domain name in each access cluster, which reduces user costs and increases users' willingness to access the distributed container cloud. There is no need to provide the kubeconfig of the access cluster for ArgoCD to create the cluster secret, reducing the risk of cluster kubeconfig leakage.

[0106] In this embodiment, a cross-network multi-cloud multi-cluster application distribution device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] This embodiment provides a cross-network multi-cloud multi-cluster application distribution device, such as Figure 8 As shown, including:

[0108] The central cluster configuration module 801 is used to create a server application and k8sAPIService resources in a pre-established central cluster, and listen to remote procedure call requests after the server application is started; the central cluster includes an API server.

[0109] The access cluster configuration module 802 is used to create a proxy application in each of the pre-established access clusters, and initiate a remote procedure call request to the API server through the proxy application.

[0110] The bidirectional long connection tunnel creation module 803 is used to forward the remote procedure call request from the proxy application to the server application through the k8s APIService resource when the API server receives it, so as to create a bidirectional long connection tunnel between the server application and the proxy application.

[0111] The application distribution module 804 is used to deploy the continuous delivery tool in the central cluster, create an access cluster key for the continuous delivery tool to distribute applications, and use the continuous delivery tool to distribute applications to multiple access clusters through a bidirectional long connection tunnel based on the access cluster key.

[0112] In some optional implementations, the central cluster configuration module 801 includes:

[0113] The central cluster configuration unit is used to create a k8s APIService resource in the pre-established central cluster and point the k8s APIService resource to the server application so that the server application can be accessed by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server.

[0114] In some optional implementations, the access cluster configuration module 802 includes:

[0115] The proxy application creation unit is used to create a proxy application including startup parameters in multiple pre-established access clusters, where the startup parameters include the APIService address of the central cluster exposed in the form of a public IP or domain name.

[0116] The request initiating unit is used to initiate a remote procedure call request to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters when the proxy application is started.

[0117] In some optional implementations, the bidirectional persistent connection tunnel creation module 803 includes:

[0118] The request forwarding unit is used to forward the remote procedure call request to the server application according to the URL path matching k8s APIService resource after the API server receives the remote procedure call request from the proxy application. When the server application receives the remote procedure call request, the server application communicates with the proxy application and creates a bidirectional long connection tunnel between the server application and the proxy application.

[0119] In some optional implementations, the application distribution module 804 includes:

[0120] The access cluster key creation unit is used to assemble the APIService address of the central cluster exposed in the form of public IP or domain name into the API server address of the access cluster, use the CA certificate of the central cluster as the CA certificate of the access cluster, use the client certificate of the central cluster as the client certificate of the access cluster, and use the client key of the central cluster as the client key of the access cluster to create the access cluster key for the continuous delivery tool application distribution.

[0121] The cluster name determination unit is used to use the name of the access cluster key as the name of the distributed access cluster when the continuous delivery tool distributes applications to multiple access clusters.

[0122] In some optional implementations, the cross-network multi-cloud multi-cluster application distribution device further includes:

[0123] The cluster creation module is used to create multiple k8s clusters, one of which is used as the central cluster, and the remaining k8s clusters except the central cluster are used as multiple access clusters; the central cluster includes the APIService address, CA certificate, client certificate and client key exposed in the form of public IP or domain name, and the APIService address exposed in the form of public IP or domain name is integrated on the API server.

[0124] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0125] The cross-network multi-cloud multi-cluster application distribution device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0126] The embodiment of the present invention also provides a computer device having the above Figure 8 A multi-cloud and multi-cluster application distribution device across a network is shown.

[0127] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0128] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0129] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0130] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0132] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0133] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0134] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0135] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0136] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A cross-network multi-cloud multi-cluster application distribution method, characterized in that: The method comprises: Create a server application and k8s APIService resources in a pre-established central cluster, and listen to remote procedure call requests after the server application is started; the central cluster includes an API server; Creating proxy applications in multiple pre-established access clusters, and initiating a remote procedure call request to the API server through the proxy applications; When the API server receives the remote procedure call request from the proxy application, it forwards it to the server application through the k8sAPIService resource to create a bidirectional long connection tunnel between the server application and the proxy application; A continuous delivery tool is deployed in the central cluster, and an access cluster key is created for use in the continuous delivery tool application distribution. Based on the access cluster key, the continuous delivery tool is used to distribute applications to multiple access clusters through a bidirectional long connection tunnel.

2. The method according to claim 1, characterized in that The central cluster and multiple access clusters are established in the following manner: Create multiple k8s clusters, use one of the k8s clusters as the central cluster, and use the remaining k8s clusters as multiple access clusters; the central cluster includes the APIService address, CA certificate, client certificate and client key exposed in the form of public IP or domain name, and the APIService address exposed in the form of public IP or domain name is integrated on the API server.

3. The method according to claim 2, characterized in that The creation of server applications and k8s APIService resources in the pre-established central cluster includes: Create a k8s APIService resource in the pre-established central cluster, and point the k8s APIService resource to the server application so that the server application can be accessed by accessing the APIService address exposed in the form of a public IP or domain name integrated into the API server.

4. The method according to claim 3, characterized in that The step of creating a proxy application in each of the pre-established multiple access clusters and initiating a remote procedure call request to the API server through the proxy application includes: Create a proxy application containing startup parameters in each of the pre-established multiple access clusters, wherein the startup parameters include the APIService address of the central cluster exposed in the form of a public IP or domain name; When the proxy application is started, a remote procedure call request is initiated to the API server through the APIService address of the central cluster exposed in the form of a public IP or domain name in the startup parameters.

5. The method according to claim 1, characterized in that When the API server receives the remote procedure call request of the proxy application, forwarding it to the server application through the k8s APIService resource to create a bidirectional long connection tunnel between the server application and the proxy application includes: When the API server receives the remote procedure call request from the proxy application, it forwards the remote procedure call request to the server application according to the URL path matching of the k8sAPIService resource. When the server application receives the remote procedure call request, the server application communicates with the proxy application and creates a bidirectional long connection tunnel between the server application and the proxy application.

6. The method according to claim 2, characterized in that Deploying the continuous delivery tool in the central cluster and creating an access cluster key for the continuous delivery tool application distribution includes: The APIService address of the central cluster exposed in the form of a public IP or domain name is assembled into the API server address of the access cluster, the CA certificate of the central cluster is used as the CA certificate of the access cluster, the client certificate of the central cluster is used as the client certificate of the access cluster, and the client key of the central cluster is used as the client key of the access cluster to create an access cluster key for use in continuous delivery tool application distribution.

7. The method according to claim 6, characterized in that The method of distributing applications to multiple access clusters through a bidirectional long connection tunnel using a continuous delivery tool based on an access cluster key includes: When the continuous delivery tool distributes applications to multiple access clusters, the name of the access cluster key is used as the name of the distributed access cluster.

8. A cross-network multi-cloud multi-cluster application distribution device, characterized in that: The device comprises: The central cluster configuration module is used to create a server application and k8sAPIService resources in a pre-established central cluster, and listen to remote procedure call requests after the server application is started; the central cluster includes an API server; An access cluster configuration module, used to create a proxy application in each of the pre-established multiple access clusters, and initiate a remote procedure call request to the API server through the proxy application; A bidirectional long connection tunnel creation module is used to forward the remote procedure call request from the proxy application to the server application through the k8s APIService resource after the API server receives it, so as to create a bidirectional long connection tunnel between the server application and the proxy application; The application distribution module is used to deploy the continuous delivery tool in the central cluster, create an access cluster key for the continuous delivery tool to distribute applications, and use the continuous delivery tool to distribute applications to multiple access clusters through a bidirectional long connection tunnel based on the access cluster key.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the cross-network multi-cloud and multi-cluster application distribution method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the cross-network multi-cloud and multi-cluster application distribution method according to any one of claims 1 to 7.

Images