Hierarchical container orchestration and container orchestration

Abstract

Embodiments of the present description provide a hierarchical container orchestration system and a container orchestration method. In the hierarchical container orchestration system, a scheduler is removed from a general untrusted container orchestration cluster and an admission controller and a custom controller are added to generate a customized container orchestration cluster, the customized container orchestration cluster is placed in a trusted execution environment to obtain a trusted container orchestration cluster, functional modification is performed on a container runtime manager in the general untrusted container orchestration cluster to obtain an untrusted container orchestration cluster, and the trusted container orchestration cluster is superimposed on the untrusted container orchestration cluster to build the hierarchical container orchestration system. During container orchestration, the untrusted container orchestration cluster and the trusted container orchestration cluster work collaboratively to achieve secure orchestration for pods; in addition, during the collaborative work, privacy protection is performed on the interaction information between the untrusted container orchestration cluster and the trusted container orchestration cluster.

Description

Hierarchical container orchestration and container orchestration Technical Field

[0001] The embodiments of this specification generally relate to the field of cloud computing, and more particularly to a hierarchical container orchestration system and a container orchestration method. Background Art

[0002] As a lightweight virtualization mechanism, containers have been widely used in cloud computing systems to efficiently deploy computing tasks on shared computing clusters. To manage containers flexibly and efficiently, efficient container orchestration is required throughout the entire container lifecycle, such as automatic restart after a crash, software update rollout, scalable scheduling, and resource allocation. Kubernetes (K8s) clusters are a popular container orchestration system for container deployment and management, particularly suitable for multi-tenant scenarios where multiple customers simultaneously lease the cluster. However, because general-purpose container orchestration systems are generally untrustworthy, using them to orchestrate containers containing sensitive data can lead to data security issues. Summary of the Invention

[0003] The embodiments of this specification provide a hierarchical container orchestration system and a container orchestration method. In the hierarchical container orchestration system, a customized container orchestration cluster is generated by removing the scheduler from the general untrusted container orchestration cluster and adding an admission controller and a customized controller, and the customized container orchestration cluster is placed in a trusted execution environment to obtain a trusted container orchestration cluster, and the container runtime manager in the general untrusted container orchestration cluster is functionally modified to obtain an untrusted container orchestration cluster, and then the trusted container orchestration cluster is superimposed on the untrusted container orchestration cluster to construct a hierarchical container orchestration system, and the constructed untrusted container orchestration cluster and the trusted container orchestration cluster work together to achieve secure orchestration for the container group. In addition, when working together, the interactive information between the untrusted container orchestration cluster and the trusted container orchestration cluster is privacy protected. Using the above-mentioned hierarchical container orchestration system, the workflow of the untrusted container orchestration cluster and the trusted container orchestration cluster can be retained to the greatest extent, and the secure orchestration of the container group can be achieved while ensuring that the container group configuration information of the trusted container group is privacy protected, while achieving unified resource management in the untrusted container orchestration cluster.

[0004] According to one aspect of an embodiment of the present specification, a hierarchical container orchestration system is provided, comprising: a trusted container orchestration cluster, comprising a trusted control plane node and a trusted data plane node, the trusted control plane node comprising a trusted API server, a trusted admission controller and a trusted data storage system, the trusted data plane node comprising a trusted container group manager, a trusted container runtime manager and a trusted container runtime; and an untrusted container orchestration cluster, comprising an untrusted control plane node and an untrusted data plane node, the untrusted control plane node comprising an untrusted API server and a container group scheduler, the untrusted data plane node comprising an untrusted container group manager, an untrusted container runtime manager and an untrusted container runtime, wherein, after intercepting a first trusted container group generated by the trusted API server in response to a received API object deployment request The trusted admission controller masks field information on the container group configuration information of the first trusted container group to obtain a shadow container group of the first trusted container group, and forwards the shadow container group to the untrusted API server for container group scheduling via the scheduler. After the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, the untrusted container runtime manager forwards the container group specification of the scheduled shadow container group to the trusted container group manager, where the container group specification includes container group configuration information and resource allocation information. The trusted container group manager pulls up the first trusted container group from the trusted API server according to the container group configuration information of the shadow container group, and uses the resources allocated based on the received resource allocation information to run the first trusted container group in the trusted container runtime.

[0005] Optionally, in an example of the above aspect, the trusted admission controller masks field information of the container group configuration information of the first trusted container group to obtain a shadow container group of the first trusted container group, including: the trusted admission controller masks field information of the container group configuration information of the first trusted container group based on a privacy protection policy to obtain a shadow container group of the first trusted container group.

[0006] Optionally, in an example of the above aspect, the trusted admission controller masks field information of the container group configuration information of the first trusted container group based on a privacy protection policy to obtain a shadow container group of the first trusted container group, including: for each field in the container configuration information of the first trusted container group, the trusted admission controller: determines whether the field is a privacy field; in response to determining that the field is a privacy field, determines whether removing the field will affect the untrusted execution in the untrusted container orchestration cluster; if it is determined that it does not affect the untrusted execution in the untrusted container orchestration cluster, removes the field from the container group configuration information to mask the field information; if it is determined that it affects the untrusted execution in the untrusted container orchestration cluster, replaces the field information of the field with insensitive data to mask the field information.

[0007] Optionally, in an example of the above aspect, the trusted admission controller masks field information of the container group configuration information of the first trusted container group to obtain a shadow container group of the first trusted container group, further comprising: if it is determined that the field is not a privacy field, determining whether the field requires other masked API objects; if no other masked API objects are required, retaining the field; if other masked API objects are required, determining whether retention or removal of the field will trigger an execution error in the untrusted container orchestration cluster and / or the trusted container orchestration cluster; if it is determined that removal of the field will trigger an execution error in the trusted container orchestration cluster, and retention of the field will not trigger an execution error in the untrusted container orchestration cluster, retaining or modifying the field information of the field; if it is determined that removal of the field will not trigger an execution error in the trusted container orchestration cluster, and / or retention of the field will trigger an execution error in the untrusted container orchestration cluster, removing the field from the container configuration information.

[0008] Optionally, in an example of the above aspect, the trusted control plane node further includes a trusted custom controller, and the trusted container group manager sends the container group configuration information of the shadow container group to the trusted API server, so that the trusted custom controller performs container group integrity verification based on the container group configuration information of the shadow container group and the container group configuration information of the first trusted container group stored in the trusted data storage system, and after the container group integrity verification is successful, the trusted container group manager pulls up the first trusted container group from the trusted API server according to the received container group configuration information.

[0009] Optionally, in an example of the above aspect, the trusted control plane node further includes a trusted custom controller, and the trusted admission controller creates a container group template for indicating scheduling constraints, and persists the container group template to the trusted data storage system. The untrusted container runtime manager further forwards the scheduling information of the first trusted container group to the trusted container group manager, and the trusted container group manager sends the scheduling information to the trusted API server to perform scheduling constraint verification according to the scheduling constraints via the trusted custom controller. After the container group integrity verification and the scheduling constraint verification are successful, the trusted container group manager pulls up the first trusted container group from the trusted API server according to the container group configuration information of the shadow container group, and uses the resources allocated based on the received resource allocation information to start the first trusted container group in the trusted container runtime.

[0010] Optionally, in an example of the above aspect, before the trusted container group manager pulls the first trusted container group from the trusted API server, an infinite lease is created for the first trusted container group.

[0011] Optionally, in an example of the above aspect, after receiving the API object deployment request, the API server converts the API object deployment request into an API object and generates a corresponding first trusted container group, and in response to the API object having data sensitivity, stores the API object and the first trusted container group in the trusted data storage system, or in response to the API object not having data sensitivity, stores the API object in an untrusted data storage system after encryption and stores the first trusted container group in the trusted data storage system.

[0012] Optionally, in an example of the above aspect, in response to receiving a first container group deletion request for deleting a first trusted container group, the trusted admission controller sends a second container group deletion request to the untrusted API server, and the trusted container group manager stops running the first trusted container group. In response to the first trusted container group stopping running, the trusted API server deletes the first trusted container group. In response to receiving the second container group deletion request, the untrusted API server deletes the shadow container group. In response to the shadow container group being deleted, the untrusted container group manager redeclares resources allocated to the shadow container group.

[0013] According to another aspect of an embodiment of the present specification, a multi-tenant container orchestration system is provided, comprising: at least two tenant container orchestration clusters, each tenant container orchestration cluster comprising a tenant trusted container orchestration cluster, the tenant trusted container orchestration cluster comprising a trusted control plane node and a trusted data plane node, the trusted control plane node comprising a trusted API server, a trusted admission controller and a trusted data storage system, the trusted data plane node comprising a trusted container group manager, a trusted container runtime manager and a trusted container runtime; and a server-side container orchestration cluster comprising a first untrusted control plane node and a first untrusted data plane node, the first untrusted control plane node comprising a first untrusted API server and a container group scheduler, the untrusted data plane node comprising a first untrusted container group manager, an untrusted container runtime manager and an untrusted container runtime, wherein, upon interception of the trusted API server After generating a first trusted container group in response to an API object deployment request initiated by a tenant via a secure communication channel, the trusted admission controller masks field information of the container group configuration information of the first trusted container group to obtain a shadow container group of the first trusted container group, and forwards the shadow container group to the untrusted API server for container group scheduling via the scheduler. After the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, the untrusted container runtime manager forwards the container group specification of the scheduled shadow container group to the trusted container group manager, where the container group specification includes container group configuration information and resource allocation information. The trusted container group manager pulls up the first trusted container group from the trusted API server and uses the resources allocated based on the received resource allocation information to run the first trusted container group in the trusted container runtime.

[0014] Optionally, in an example of the above aspect, the first untrusted control plane node further includes a synchronizer, and the untrusted API server communicates with the trusted admission controller via the synchronizer.

[0015] Optionally, in an example of the above aspect, the tenant container orchestration cluster further includes a tenant untrusted container orchestration cluster located between the tenant trusted container orchestration cluster and the server-side container orchestration cluster, the tenant untrusted container orchestration cluster including a second untrusted control plane node and a second untrusted data plane node, the second untrusted control plane node including a second untrusted API server, and the second untrusted data plane node including a second untrusted container group manager. The trusted admission controller and the first untrusted API server communicate via the second untrusted API server, and the untrusted container runtime manager sends the container group specification and resource allocation information of the scheduled first trusted container group to the trusted container group manager via the second untrusted container group manager.

[0016] According to another aspect of an embodiment of the present specification, a container orchestration method is provided, comprising: in response to a trusted API server in a trusted container orchestration cluster receiving an API object deployment request, converting the API object deployment request into a requested API object via the trusted API server and generating a corresponding first trusted container group, and storing the first trusted container group in a trusted data storage system in the trusted container orchestration cluster; in response to a trusted admission controller in the trusted container orchestration cluster intercepting the first trusted container group, masking field information of container group configuration information of the first trusted container group via the trusted admission controller to obtain a first trusted container group. The method further comprises: pulling up the shadow container group from the untrusted API server and forwarding the shadow container group to the untrusted API server for container group scheduling via the scheduler; after the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, forwarding the container group specification of the scheduled shadow container group to the trusted container group manager via the untrusted container runtime manager, wherein the container group specification includes container group configuration information and resource allocation information; pulling up the first trusted container group from the trusted API server via the trusted container group manager, and running the first trusted container group in the trusted container runtime using the resources allocated based on the received resource allocation information.

[0017] Optionally, in an example of the above aspect, the container orchestration method may further include: in response to receiving a first container group deletion request for deleting a first trusted container group, sending a second container group deletion request to the untrusted API server via the trusted admission controller, and stopping the operation of the first trusted container group via the trusted container group manager; in response to the first trusted container group stopping operation, deleting the first trusted container group via the trusted API server; in response to receiving the second container group deletion request, deleting the shadow container group via the untrusted API server, and in response to the shadow container group being deleted, re-declaring resources allocated to the shadow container group via the untrusted container group manager.

[0018] According to another aspect of an embodiment of this specification, a container orchestration system is provided, comprising: at least one processor; a memory coupled to the at least one processor; and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the container orchestration method described above.

[0019] According to another aspect of an embodiment of this specification, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the container orchestration method described above.

[0020] According to another aspect of an embodiment of this specification, a computer program product is provided, including computer program instructions, where the computer program instructions are executed by a processor to implement the container orchestration method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] A further understanding of the nature and advantages of the present disclosure may be achieved by referring to the following drawings, in which similar components or features may have the same reference numerals.

[0022] Figure 1 shows an example schematic diagram of a general K8s cluster.

[0023] FIG2 shows a schematic diagram of an example architecture of a hierarchical container orchestration system according to an embodiment of this specification.

[0024] FIG3 shows an example schematic diagram of a Pod creation process according to an embodiment of this specification.

[0025] FIG4 shows an example flowchart of a field masking process of a trusted Pod according to an embodiment of this specification.

[0026] FIG5 shows an example schematic diagram of a trusted Pod according to an embodiment of the present specification.

[0027] FIG6 shows an example schematic diagram of a shadow Pod according to an embodiment of this specification.

[0028] FIG7 shows an exemplary schematic diagram of a scheduling constraint verification process according to an embodiment of this specification.

[0029] FIG8 shows an example schematic diagram of a Pod deletion process according to an embodiment of this specification.

[0030] FIG9 shows a schematic diagram of an example architecture of a multi-tenant container orchestration system according to an embodiment of this specification.

[0031] FIG10 shows another example architecture diagram of a multi-tenant container orchestration system according to an embodiment of this specification.

[0032] FIG11 shows an example schematic diagram of a hierarchical container orchestration system implemented based on a computer system according to an embodiment of this specification. DETAILED DESCRIPTION

[0033] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope of protection, applicability, or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of this specification. Various examples may omit, replace, or add various processes or components as needed. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described relative to some examples may also be combined in other examples.

[0034] As used herein, the term "including" and its variations are open terms meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0035] The flowcharts used in this specification illustrate operations implemented by systems according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Rather, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0036] A container is a group of processes isolated from the rest of the host system. The Linux kernel supports namespaces to achieve this isolation and control groups (e.g., cgroups) to limit the resources used by a process group. To ensure portability of containers between different hosts, the Open Container Initiative (OCI) has created open standards, including runtime-spec (which defines the container's behavior), image-spec (which defines the image format), and distribution-spec (which defines push to a registry). High-level container runtimes (e.g., Docker, containerd, and CRIO) implement image-spec and distribution-spec to manage the entire container lifecycle. For example, they pull an OCI image from a registry and then unpack it into an OCI runtime filesystem bundle, which includes the container's root filesystem and a configuration file called config.json. They then call low-level container runtimes (e.g., RunC and Kata) to create processes using this filesystem bundle, which implements the runtime-spec. Unlike RunC, which is based on Linux namespaces and groups, hardware virtualization technology can be used for container isolation, thereby providing stronger workload isolation.

[0037] As container scale increases, manual container management becomes inconvenient, necessitating the need for container orchestration systems. These systems can, for example, self-heal crashed containers, automatically roll out and roll back container updates, and automatically schedule containers between nodes. The Kubernetes cluster (Kubernetes architecture) is the mainstream framework for deploying and managing containers and has become the standard for container orchestration.

[0038] Replacing containers, the smallest deployment unit in a k8s cluster is a Pod. A Pod is a collection of containers (container group) that share a network namespace (same IP address and port space), an IPC namespace, and storage volumes. A Pod can contain one or more containers, which are usually closely collaborating application components that share the same lifecycle and resources. The K8s cluster schedules and manages the Pod as a whole, which means that the containers in the Pod always run on the same node and can communicate efficiently locally.

[0039] The K8s API is a resource-based API (RESTful API), and Pod is the basic API resource type. Pods are deployed directly by creating Pod objects using the k8s command-line tool called kubectl, which is often difficult to manage. To this end, advanced workload resources such as Deployments, StatefulSets, and Jobs can be deployed through kubectl. These advanced workload resources usually represent a set of heterogeneous Pods. The K8s cluster automatically manages these Pods, for example, creating new Pods for crash recovery. In addition, the K8s cluster can also provide many other resources, such as ConfigMaps and Secrets, which contain configuration information and sensitive metadata used by Pods. In this manual, API resources can also be referred to as API objects.

[0040] Figure 1 shows an example schematic diagram of a general K8s cluster.

[0041] As shown in Figure 1, a general Kubernetes cluster includes a control plane node and one or more data plane nodes. Each control plane node and data plane node can be composed of one or more data processing devices or apparatuses. The control plane node includes an API server, a state controller, a scheduler, and an Etcd data storage system. The data plane node includes the Kubelet container group manager, the CRI shim container runtime manager, the Container Runtime, and the Kube-proxy network proxy, and runs the launched Pods in the Container Runtime. In some embodiments, the CRI shim container runtime manager can also be integrated with the Container Runtime.

[0042] The API server is the main interface of the K8s cluster, and all operations and management of Pods are performed through the API server. The API server exposes a set of RESTful APIs and allows users to interact with other components in the K8s cluster through this set of APIs. After receiving the API request entered by the user through the client (for example, using the command line tool kubectl), the API server converts the received API request into an internal API object and forwards the API object to the corresponding K8s component for processing. The API server is also responsible for authentication, authorization, and identity verification to ensure that only authorized users and components can access and operate the K8s cluster.

[0043] The state controller represents a control loop that continuously monitors the current state of the K8s cluster and is used to manage and control the state of various resources in the K8s cluster. The state controller ensures that the desired state is consistent with the current state by continuously monitoring changes in the state of resources in the K8s cluster. Examples of state controllers may include, but are not limited to, Deployment controllers, ReplicaSet controllers, StatefulSet controllers, etc. The Deployment controller is used to manage the deployment of applications, the ReplicaSet controller is used to ensure that a specified number of Pod replicas are running, and the StatefulSet controller is used to manage the deployment of stateful applications. For example, after the API server receives the StatefulSet object and stores it in the data storage system Etcd, the StatefulSet controller that monitors the data storage system Etcd discovers that a new StatefulSet object has been created, and the StatefulSet controller creates the corresponding Pod (Pod object) through the API server.

[0044] The scheduler is responsible for assigning pods to nodes in the Kubernetes cluster. Based on a variety of scheduling policies and the pod's resource requirements, the scheduler can select the most appropriate node to run the pod, for example, by binding the pod to the selected node. The scheduler monitors the resource utilization and load of nodes in the Kubernetes cluster and makes intelligent scheduling decisions based on predefined policies.

[0045] The data storage system Etcd is a distributed key-value storage system in the K8s cluster, used to store the status and metadata of the K8s cluster. Etcd is a non-volatile data storage system that provides high availability, consistency, and persistence, and can quickly respond to read and write requests. Various components and controllers in the K8s cluster use Etcd to store and read the configuration information, status information, and event notifications of the K8s cluster, such as Pod configuration information, Pod scheduling constraints, etc. Etcd can be deployed on the control plane node or as a separate cluster. When deployed as a separate cluster, metadata will not be lost even if the entire K8s cluster is restarted.

[0046] The container group manager Kubelet includes various managers for the entire life cycle of the Pod, which is used to monitor the creation or deletion of the Pod and run or stop the Pod accordingly. The running or stopping of the Pod depends on the communication with the container runtime via the container runtime interface (CRI). CRI is a plug-in interface that allows the container manager Kubelet to interact with various container runtimes without recompilation. There is a CRI shim layer between the container manager Kubelet and the container runtime, for example, Containerd's CRI. The network agent Kube-proxy is used to maintain network rules for communication between the outside and inside of the K8s cluster and the Pod.

[0047] In cloud computing systems, while general-purpose Kubernetes clusters are easy to deploy and offer flexible scalability, they are generally untrustworthy, raising data privacy concerns when sensitive data must be outsourced to third-party public cloud platforms for processing. To implement trusted computing in Kubernetes clusters, a solution combining trusted execution environments (TEEs) with Kubernetes clusters has been proposed.

[0048] A trusted execution environment can provide a trusted area in an untrusted server, and the trusted area is isolated from the untrusted area using hardware mechanisms. A trusted area can be a process (e.g., a trusted boundary (enclave) in Intel SGX) or a virtual machine (e.g., a trusted domain (TD) in Intel TDX). TEE provides clients with a verification mechanism to verify that the expected code has been loaded into the trusted area. Potentially malicious privileged software such as the host operating system (OS) or VM manager cannot compromise the confidentiality and integrity of data and code in the trusted area. In a trusted area, only volatile states (in the CPU or memory) can be protected, while non-volatile storage information is generally not protected. Examples of commercial CPUs that support TEE may include, but are not limited to: ARM TrustZone, Intel SGX, AMD SEV, ARM CCA, Intel TDX, etc.

[0049] Trusted execution environments (TEEs) can include, for example, process-based TEEs and virtual machine (VM)-based TEEs. In a process-based TEE, applications are ported to the TEE. In a VM-based TEE, the entire VM is isolated and protected, allowing unmodified applications to run directly within the VM. Compared to process-based TEEs, VM-based TEEs make it easier to build trusted I / O, thus facilitating heterogeneous computing. Therefore, VM-based TEEs are preferred in cloud computing, especially for cloud-native applications using containers.

[0050] VM-based TEEs allow applications to run unmodified. When applying VM-based TEEs to Kubernetes clusters, there are both integrated and minimal protection options. In an integrated protection solution, such as Constellation, a VM TEE is used to encapsulate each Kubernetes cluster running sensitive workloads. With this protection solution, in multi-tenant scenarios, different Constellation instances are launched for different tenants. For each tenant cluster, the control plane and data plane are encapsulated within the VM TEE. Using different VM TEEs for different Kubernetes clusters separates control planes, which cannot easily communicate with each other, leading to suboptimal resource scheduling. Because nodes are the granularity of leased hardware resources in this leasing architecture, tenants need to lease larger nodes for future workloads, forcing the cloud provider to sell its limited resources to fewer tenants, resulting in underutilized resources for the cloud provider and unnecessary costs for tenants.

[0051] In minimal protection schemes, such as Confidential Containers (CoCo), pods are protected by simply residing in their own VM TTEs. Because pods run workloads involving sensitive data, and the Kubernetes framework does not directly process this sensitive data, minimal protection schemes are considered sufficiently secure. However, minimal protection schemes are susceptible to various attacks, such as replacing VM images, tampering with container images, and modifying client agents, making them difficult to address without major modifications to the Kubernetes architecture.

[0052] In view of the above, an embodiment of the present specification provides a hierarchical container orchestration system (container orchestration framework). In the hierarchical container orchestration system, a customized container orchestration cluster is generated by removing the scheduler from the general untrusted container orchestration cluster and adding an admission controller and a custom controller, and the customized container orchestration cluster is placed in a trusted execution environment to obtain a trusted container orchestration cluster, and then the trusted container orchestration cluster is superimposed on the general untrusted container orchestration cluster to construct a hierarchical container orchestration system, and the untrusted container orchestration cluster and the trusted container orchestration cluster work together to achieve secure orchestration of the container group. In addition, when working together, the interactive information between the untrusted container orchestration cluster and the trusted container orchestration cluster is privacy protected. By using the above-mentioned hierarchical container orchestration system, the workflow of the untrusted container orchestration cluster and the trusted container orchestration cluster can be retained to the greatest extent, and the secure orchestration of the container group can be achieved while ensuring that the container group configuration information of the trusted container group is privacy protected, while achieving unified resource management in the untrusted container orchestration cluster.

[0053] The following describes the hierarchical container orchestration system and container orchestration method according to the embodiments of this specification using the K8s framework as an example. It should be noted that in other embodiments, other container orchestration frameworks can also be used to implement the hierarchical container orchestration system described in this specification.

[0054] FIG2 shows a schematic diagram of an example architecture of a hierarchical container orchestration system according to an embodiment of this specification.

[0055] As shown in Figure 2, the hierarchical container orchestration system includes an untrusted K8s cluster and a trusted K8s cluster, and the trusted K8s cluster is superimposed on the untrusted K8s cluster.

[0056] The untrusted k8 cluster includes untrusted control plane nodes and untrusted data plane nodes. The untrusted control plane nodes include an untrusted API server and a container group scheduler, and the untrusted data plane nodes include an untrusted container group manager Kubelet and an untrusted container runtime Container Runtime. Between the untrusted container group manager Kubelet and the untrusted container runtime Container Runtime, there exists an untrusted container runtime manager CRI Shim, for example, Contained's CRI Containerd. In this specification, the CRI shim can be obtained by modifying the CRI Shim in a general K8s cluster. In addition to retaining the original functions, the modified CRI Shim can also perform trusted Pod identification based on Pod configuration information, and after identifying the trusted Pod, forward the received Pod configuration information and resource allocation information to the trusted container manager delegated Kubelet in the trusted K8 cluster.

[0057] In addition, the untrusted K8s cluster can also include an untrusted data storage system Etcd (untrusted Etcd). Untrusted Etcd is used to store the status and metadata of the untrusted K8s cluster, such as the Pod configuration information of the Pod scheduled by the scheduler.

[0058] In some embodiments, the untrusted K8s cluster can be obtained by modifying the general K8s cluster. For example, the untrusted K8s cluster can be achieved by modifying the CRI Shim in the general K8s cluster while keeping other components of the general K8s cluster unchanged.

[0059] The trusted K8s cluster includes a trusted control plane node and a trusted data plane node. The trusted control plane node includes a trusted API server, a trusted admission controller, a trusted custom controller, and a trusted data storage system Etcd, and the trusted data plane node includes a trusted container group manager Kubelet, a trusted container runtime manager, and a trusted container runtime. In some embodiments, a trusted K8s cluster can be implemented by adding an admission controller and a custom controller to the control plane node of a general K8s cluster and removing the scheduler, and then placing the modified K8s cluster in a trusted execution environment. In one example, for example, the scheduler in the K8s cluster can be removed by terminating the binary file running the scheduler.

[0060] After completing the component modifications for the general K8s cluster as described above, in order to bootstrap the trusted K8s cluster, a set of VM TEEs can be started to act as master nodes and worker nodes, and the user can verify by providing the measurement values ​​of the trusted zone construction process and the initial content of the trusted zone (including the OS kernel and K8s components). After the user completes the verification, a secure channel can be established between the user's client device and all VM TEEs, and then the integer and password are distributed to the components in the trusted K8s cluster to establish an intra-component TLS connection, thereby completing the creation and startup of the trusted K8s cluster.

[0061] A trusted admission controller intercepts API requests from client devices, validates their behavior, and modifies the API objects in the requests based on certain rules. For example, it can populate missing fields in the API objects with default values. Furthermore, trusted admission controllers can be used to create validation and modification rules, such as scheduling constraints. In the Kubernetes framework, trusted admission controllers can be implemented using admission webhooks.

[0062] Trusted custom controllers are used to implement validation of API request processing results, such as Pod integrity verification and Pod scheduling constraint verification during Pod creation requests. Trusted Etcds are used to persist API objects and Pod objects created by the API server. It should be noted that in some embodiments, trusted control plane nodes may not include trusted custom controllers, thereby omitting Pod integrity verification and Pod scheduling constraint verification during Pod creation.

[0063] The trusted container group manager Kubelet is used to pull trusted Pods from the trusted API server and run the pulled trusted Pods in the trusted container runtime.

[0064] After establishing a secure communication channel between the trusted API server and the user's client device, the user can use the client device (for example, the command-line tool Kubectl) to send various API requests to the trusted API server via the secure communication channel, such as API object deployment requests and Pod deletion requests. Subsequently, the untrusted Kubernetes cluster and the trusted Kubernetes cluster can work together to process the API requests, thereby implementing container orchestration, such as Pod creation and deletion.

[0065] FIG3 shows an example schematic diagram of a Pod creation process according to an embodiment of this specification.

[0066] As shown in Figure 3, after receiving an API object deployment request sent by a user using the command-line tool Kubectl, the trusted API server converts the received API object deployment request into the requested API object and generates a corresponding first trusted Pod (also called a Pod object). API objects are abstract entities used to describe and operate different resources in a K8s cluster and are the basic units for create, update, and delete operations performed through the K8s API. API objects define the specifications, status, and metadata of resources. By interacting with API objects, operations such as automated resource management, expansion, and fault recovery can be implemented, thereby improving the reliability and manageability of container applications.

[0067] In some embodiments, if the requested API object is a low-level API object (low-level work resource), such as a ConfigMap, the trusted API server can convert the received API request into an internal low-level API object according to internal rules, and for the API object that will generate a Pod, generate the corresponding first trusted Pod according to the built-in logic. If the requested API object is a high-level API object (high-level work resource), such as a StatefulSet, the trusted API server can convert the received API request into an internal high-level API object according to internal rules, and for the API object that will generate a Pod, generate the corresponding first trusted Pod according to the built-in logic via the corresponding state controller (e.g., a StatefulSet controller). In some embodiments, the generated API object and Pod can be persisted to a trusted Etcd. In some embodiments, based on the object diversion strategy, in response to the converted API object having data sensitivity, the API object and the first trusted Pod can be persisted to a trusted Etcd, and in response to the converted API object not having data sensitivity, the API object can be encrypted and persisted to an untrusted Etcd, and the first trusted Pod can be persisted to a trusted Etcd.

[0068] Trusted Etcd carries other overhead, such as encryption and authentication of persistent data, freshness protection using Merkle trees, and invoking trusted counter services. Therefore, access to trusted Etcd needs to be reduced. To this end, an object diversion strategy based on data sensitivity is proposed.

[0069] According to the object diversion strategy, only API objects with data sensitivity and the first trusted Pod are persisted to the trusted Etcd. For API objects that are not data sensitive, such as the resource specification API and the operation & maintenance API, since there is no sensitive data, they do not need to be persisted to the trusted Etcd, but are persisted to the untrusted Etcd after encryption. Examples of resource specification APIs may include but are not limited to ResourceQuotaSpec (used to set quota limits for each namespace) and PersistentVolumeClaim (request and declare persistent volumes). Examples of operation & maintenance APIs may include but are not limited to Event (recording events in the cluster in a best-effort manner). Based on the object diversion strategy, the built-in etcd-servers-overrides configuration of the API server can be used to set the destinations of different API objects to different Etcd clusters.

[0070] In some embodiments, in order to avoid deploying another untrusted Etcd cluster for the trusted K8s cluster, the untrusted Etcd in the untrusted K8s cluster can be reused and distinguished from the untrusted API objects by setting the etcd-prefix string flag. In this way, all insensitive API objects in the trusted K8s cluster can be placed in a directory with a prefix path, thereby avoiding potential conflicts.

[0071] After intercepting the generated first trusted Pod, the trusted admission controller masks the first trusted Pod's Pod configuration information to generate a shadow Pod. The shadow Pod is then forwarded to the untrusted API server in the untrusted Kubernetes cluster for scheduling by the scheduler. The shadow Pod is a desensitized resource consumption representation of the Pod object in the trusted Kubernetes cluster managed by the untrusted container group manager.

[0072] Pod configuration information contains a variety of fields, including, for example, fields related to containers, volumes, service accounts, host names, and more. Some of these fields contain private information, others contain information related to API object references and resource consumption, and some contain information related to Pod orchestration configuration or processing.

[0073] The generated Shadow Pod should ensure that: (1) all private information is removed or replaced with insensitive data; (2) no unexpected errors are triggered during the life cycle of the Shadow Pod (for example, if the trusted Pod references other API objects in the trusted K8s cluster and the Shadow Pod retains these references, resource discovery errors will be triggered because these API objects do not exist in the untrusted K8s cluster); (3) the Shadow Pod itself occupies minimal resources (for example, if the trusted Pod requires an image and the image field of the Shadow Pod does not change, unnecessary storage usage will occur).

[0074] In some embodiments, the trusted admission controller may perform field masking on the Pod configuration information of the first trusted Pod based on a privacy protection policy. In some embodiments, the trusted admission controller may also perform field masking on the Pod configuration information of the first trusted Pod based on a resource minimization policy and / or an unexpected error trigger avoidance policy.

[0075] Figure 4 shows an example flow chart of a trusted Pod field masking process 400 according to an embodiment of the present disclosure. It should be noted that the field masking process shown in Figure 4 is a process for processing a single field in the trusted Pod configuration information. When performing field masking, the field masking process shown in Figure 4 needs to be executed for all fields in the Pod configuration information.

[0076] As shown in Figure 4, at 401, determine whether the data recorded by the current Pod field is private data, that is, whether the current Pod field is a private field. If it is determined that the current Pod field is a private field, then at 402, determine whether removing the current Pod field will affect the untrusted execution in the untrusted K8s cluster. If it is determined that it will not affect the untrusted execution in the untrusted K8s cluster, then at 403, the field information is masked by removal. For example, for the .spec.container[].env[] field containing sensitive values ​​of environment variables, since its value is empty, it will not affect the untrusted execution in the untrusted K8s cluster, and thus the field information can be masked by removal. If it is determined that it affects the untrusted execution in the untrusted K8s cluster, then at 404, the field information is masked by replacing insensitive data. For example, for the required.spec.container[].image field used to specify the image to be pulled, since its value is empty, it will affect untrusted execution in an untrusted K8s cluster. Therefore, it is necessary to use a pseudo field (for example, k8s.gcr.io / pause) to replace the field information to mask it.

[0077] If it is determined that the current Pod field is not a private field, then at 405, it is determined whether the current Pod field requires other shadowed API objects (Shadow objects), that is, Shadow object references. If no Shadow object references exist, such as .spec.hostname and .spec.container[].resource, then at 406, the current Pod field is retained so that the untrusted kubelet can consider the resources occupied by the current trusted Pod.

[0078] If there are Shadow references, such as the ConfigMaps field, the ServiceAccounts field, and the .smetadata.namespace field, then in 407 , determine whether retaining or removing the current Pod field will trigger an execution error in the untrusted K8s cluster / trusted K8s cluster.

[0079] If it is determined that removing the current Pod field will trigger an execution error in the trusted K8s cluster, and retaining the current Pod field will not trigger an execution error in the untrusted K8s cluster, then at 408, retain or modify the current Pod field. For example, the .smetadata.namespace field in the trusted Pod is used to distinguish Pods with the same name, so these fields should be retained in the shadow Pod. Otherwise, shadow Pods with the same name will be merged. For the namespece field, since the namespece field corresponds to the Namespace object, the Namespce object is also required to be masked, so the field value in the namespace field needs to be modified.

[0080] If it is determined that removing the current Pod field will not trigger an execution error in the trusted Kubernetes cluster, and / or retaining the current Pod field will trigger an execution error in the untrusted Kubernetes cluster, then the current Pod field is removed in step 409. For example, the trusted Kubernetes cluster does not need the untrusted execution results of the ConfigMaps and ServiceAccounts fields in the untrusted Kubernetes cluster, so the ConfigMaps and ServiceAccounts fields can be removed.

[0081] FIG. 5 shows an example schematic diagram of a trusted Pod according to an embodiment of this specification, and FIG. 6 shows an example schematic diagram of a shadow Pod according to an embodiment of this specification.

[0082] Furthermore, for confidentiality and resource management reasons, the number of fields removed or replaced needs to be limited. Furthermore, a key-value pair, in.metdata.annotations, can be added to indicate that the Pod is a Shadow Pod, so that the container runtime manager (CRI) shim will not pull up the image in an untrusted Kubernetes cluster. It should be noted that for volumes such as local, since they do not store sensitive data, untrusted Kubernetes clusters should consider the resources they consume, and therefore these fields should remain in the Shadow Pod.

[0083] After receiving the shadow Pod, the untrusted API server persists the shadow Pod to an untrusted Etcd and provides the shadow Pod to the scheduler for container group scheduling. The scheduler monitors the node resource utilization and workload in the untrusted Kubernetes cluster and, based on a series of scheduling policies and the Pod resource requirements defined in the Pod configuration information, schedules the shadow Pod to the most appropriate data plane node for execution. For example, the scheduler can schedule the shadow Pod to an available data plane node based on the shadow Pod's resource requirements, affinity, and anti-affinity policies.

[0084] After a shadow pod is scheduled to an available data plane node, the untrusted Kubelet on that node pulls the shadow pod from the untrusted API server and allocates resources for it. After retrieving the shadow pod's pod configuration and resource allocation information from the untrusted Kubelet, the CRI shim forwards the shadow pod's pod configuration and resource allocation information to the trusted Kubelet. These pod configuration and resource allocation information are collectively referred to as the pod specification.

[0085] After receiving the Pod specification of the shadow Pod, the trusted Kubelet sends the Pod configuration information in the Pod specification of the shadow Pod to the trusted API server. The trusted Kubelet pulls up the first trusted Pod from the trusted API server according to the Pod configuration information of the shadow Pod, and allocates resources to the first trusted Pod based on the received resource allocation information. Subsequently, the allocated resources are used to start the first trusted Pod in the trusted container runtime. For example, the trusted Kubelet can use the Pod name and Pod namespace in the Pod configuration information of the shadow Pod to pull up the first trusted Pod from the trusted API server. Then, the received resource allocation information (i.e., hardware resource allocation information) is filled in when the first trusted Pod starts, thereby using the allocated resources to start the first trusted Pod.

[0086] In some embodiments, the optional control plane node may include a trusted custom controller. The trusted API server may forward the Pod configuration information of the shadow Pod to the trusted custom controller. The trusted custom controller retrieves the Pod configuration information of the first trusted Pod from the trusted Etcd, and uses the retrieved Pod configuration information of the first trusted Pod and the received Pod configuration information of the shadow Pod to perform container group integrity verification to verify whether Pod tampering occurs during the scheduling process. In some embodiments, the trusted custom controller may use the Pod definition field in the Pod configuration information of the shadow Pod to perform container group integrity verification. For example, the trusted custom controller may extract the Pod name and Pod namespace from the Pod definition field in the Pod configuration information of the shadow Pod, and perform container group integrity verification based on the extracted Pod name and Pod namespace.

[0087] After the integrity of the container group is successfully verified, the trusted Kubelet pulls up the first trusted Pod from the trusted API server based on the Pod configuration information of the shadow Pod and allocates resources to the first trusted Pod based on the received resource allocation information. Subsequently, the first trusted Pod is started in the trusted container runtime using the allocated resources. For example, the trusted Kubelet can use the Pod name and Pod namespace in the Pod configuration information of the shadow Pod to pull up the first trusted Pod from the trusted API server. Then, when the first trusted Pod is started, the received resource allocation information (i.e., hardware resource allocation information) is filled in, thereby starting the first trusted Pod using the allocated resources.

[0088] In the Pod specification of the shadow Pod received from the untrusted K8s cluster, the trusted Kubelet mainly uses the following field information in the Pod specification: (1) the Pod name and namespace field indicating the Pod to be started, and (2) the resource allocation information field used to indicate the path to allocate resources (such as volumes). Considering that the first field information comes from the untrusted K8s cluster, it may deceive the trusted Kubelet into running the wrong Pod. It should be noted that although the Pod to be run is pulled directly from the trusted API server by the trusted Kubelet, which can ensure the integrity of a single Pod, the scheduler is deployed in the untrusted K8s cluster, so that the integrity of the Pod orchestration may still be compromised, for example, starting two Pods with the same name or starting Pods in an unexpected number, because there is no trusted centralized component to ensure that the same Pod can only be assigned to one node. In addition, each trusted Kubelet will try to pull the Pod independently when receiving the Pod configuration, which may generate two identical Pods on different nodes.

[0089] To this end, scheduling constraints can be created in the admission controller to solve the above problems. For example, a custom CRD object in the K8s framework can be used to define an additional resource object (for example, a Pod template) in the admission controller as a scheduling constraint (Scheduling Constraint) that needs to be followed when scheduling the Pod. For example, scheduling constraints can be imposed on the deployment relationship of multiple pods for advanced resources (advanced workloads). The created container group template is persisted to the trusted Etcd, so that when the trusted Kubelet pulls up the Pod, the container group template stored in the trusted Etcd can be used to verify the scheduling constraints.

[0090] The following uses the StatefulSet object as an example to illustrate scheduling constraints. The scheduling constraints of the StatefulSet object mainly include the following fields: (1) StatefulSet hash; (2) Whether it is root; (3) Number of Pods; (4) Pod map: id->(hash, allow_start).

[0091] When a trusted admission controller intercepts a CRUD request for a higher-level resource, it needs to create or delete a new scheduling constraint. When creating a scheduling constraint, fill in the constraint fields based on the higher-level resource's content, leaving fields like Pod names untouched for later Pod interception.

[0092] When scheduling constraints are first generated, the first three fields are filled in. When a Pod is generated, if the parent hash in the Pod matches the StatefulSet hash in the Pod template, the Pod map is filled with the value of specific_id->(specific_hash, false).

[0093] Before the trusted Kubelet starts a pod, the trusted custom controller uses the pod's ID to index and compares the hash value. If the comparison is successful, it sets allow_start to true. After the trusted Kubelet detects that allow_start is true, it allows the pod to start.

[0094] Furthermore, even after creation, advanced resources are still subject to update operations, which can result in the addition, reduction, or restart of Pods. For native resources in the K8s framework, whether it's a Deployment, StatefulSet, or the Pod itself, if you want to upgrade the image in the Pod, K8s will destroy the Pod and reschedule and create a new Pod. For StatefulSet, although the original Pod name can be retained, the actual UID and Pod IP will change. Based on the above considerations, it is necessary to intercept the update operation of advanced resources.

[0095] After intercepting the update of advanced resources, first, index the corresponding scheduling constraints based on the Pod name and Pod namespace, and take out the hash value of the Pod template filled in when it was created and the hash value of the entire object to determine the updated content. If it is determined that the Pod template has been updated, it is necessary to recreate the scheduling constraints, that is, delete the old scheduling constraints and create new scheduling constraints. If the updated field is outside the Pod template, for example, the replica field is modified, no operation is performed, and the webhook-pod will update the scheduling constraints. When intercepting the pod generated by the advanced resources, if the scheduling constraints do not include this pod, then the pod will be added to the scheduling constraints. If the corresponding pod already exists, it will be updated with the latest content.

[0096] If scheduling constraints exist, the untrusted CRI Shim can further forward the scheduling information of the second trusted container group to the trusted Kubelet. The trusted Kubelet then sends the received scheduling information to the trusted API server, which forwards it to the trusted custom controller for scheduling constraint verification based on the scheduling constraints stored in the trusted Etcd.

[0097] FIG7 shows an exemplary schematic diagram of a scheduling constraint verification process according to an embodiment of this specification.

[0098] As shown in Figure 7, a Pod template is created in the trusted admission controller to indicate scheduling constraints. This template is then persisted as an API object in the trusted Etcd. Pod templates define the policies that API objects (API resources) must adhere to. For example, for a StatefulSet object, a scheduling constraint can be set such that "Server-id cannot contain multiple MySQL-0 instances, thus preventing multiple primary servers."

[0099] After receiving an API object deployment request, the trusted API server creates a corresponding trusted Pod. After generating a shadow Pod for the created trusted Pod via the trusted admission controller, the shadow Pod is forwarded to the scheduler in the control plane node of the untrusted Kubernetes cluster for Pod scheduling. The untrusted Kubelet transmits the Pod configuration and scheduling information of the scheduled shadow Pod back to the trusted Kubelet via the untrusted CRI Shim. The trusted Kubelet returns the scheduling information to the trusted API server for forwarding to the trusted custom controller. The trusted custom controller uses the Pod template in the trusted Etcd and the received Pod configuration and scheduling information to verify the container group integrity and scheduling constraints.

[0100] After the container group integrity and scheduling constraint verification are successful, the trusted Kubelet pulls the first trusted Pod from the trusted API server based on the shadow Pod's Pod information and allocates resources to the first trusted Pod based on the received resource allocation information. It then uses the allocated resources to start the first trusted Pod in the trusted container runtime.

[0101] In some embodiments, before the trusted Kubelet pulls the first trusted Pod from the trusted API server, an indefinite lease can be created for the first trusted Pod. In some embodiments, an indefinite lease is attempted only when the Pod configuration for pulling the first trusted Pod comes from an untrusted CRI shim, thereby eliminating the opportunity for incorrectly occupying the leader node, thereby improving security and performance. This indefinite lease can be implemented using the Lease object in the Kubernetes framework.

[0102] FIG8 shows an example schematic diagram of a Pod deletion process according to an embodiment of this specification.

[0103] As shown in Figure 8, in response to receiving the first Pod deletion request for deleting the first trusted Pod, the trusted API server marks the first trusted Pod as "Deleting." After the trusted Kubelet detects that the first trusted Pod has been marked for deletion, it stops running the first trusted Pod and notifies the trusted API server of the decommissioning. After receiving the notification that the first trusted Pod has stopped running, the API server deletes the first trusted Pod stored in the trusted Etcd.

[0104] After intercepting the first Pod deletion request, the trusted admission controller sends a second Pod deletion request to the untrusted API server. In response to receiving the second Pod deletion request, the untrusted API server deletes the shadow Pod. After detecting that the shadow Pod has been deleted, the untrusted Kubelet reclaims the resources allocated to the shadow Pod, thereby releasing the API resources occupied by the shadow Pod. According to this Pod deletion method, the trusted Pod deletion process in the trusted K8s cluster does not rely on any components in the untrusted K8s cluster, thereby achieving safe deletion of the trusted Pod.

[0105] The hierarchical container orchestration system described above with reference to Figures 1 to 8 is a container orchestration system applied to a single-tenant scenario. It should be noted that in a multi-tenant scenario, the container orchestration system shown in Figure 2 needs to be modified.

[0106] FIG9 shows a schematic diagram of an example architecture of a multi-tenant container orchestration system according to an embodiment of this specification.

[0107] As shown in FIG9 , a multi-tenant container orchestration system includes at least two tenant container orchestration clusters and a server-side container orchestration cluster. Each tenant container orchestration cluster includes a tenant trusted container orchestration cluster. The tenant trusted container orchestration cluster may include a trusted control plane node and a trusted data plane node. The trusted control plane node includes a trusted API server, a trusted admission controller, a trusted customization controller, and a trusted data storage system, and the trusted data plane node includes a trusted Kubelet, a trusted container runtime manager, and a trusted container runtime. The structure and operation of the tenant trusted container orchestration cluster can refer to the structure and operation of the trusted container orchestration cluster described above with reference to FIG1 to FIG8 .

[0108] The server-side container orchestration cluster includes a first untrusted control plane node and a first untrusted data plane node. The first untrusted control plane node includes a first untrusted API server and a container group scheduler, and the untrusted data plane node includes a first untrusted Kubelet, an untrusted container runtime manager, and an untrusted container runtime. The structure and operation of the server-side trusted container orchestration cluster can refer to the structure and operation of the untrusted container orchestration cluster described above with reference to Figures 1 to 8.

[0109] In a multi-tenant container orchestration system, a separate tenant container orchestration cluster can be customized for each tenant, and the customized tenant container orchestration cluster can be superimposed on the server container orchestration cluster in a pluggable manner, thereby achieving flexible configuration of tenant clusters.

[0110] Figure 10 shows another exemplary architecture diagram of a multi-tenant container orchestration system according to an embodiment of this specification. The multi-tenant container orchestration system shown in Figure 10 is a modified embodiment of the multi-tenant container orchestration system shown in Figure 9 .

[0111] Compared with the embodiment shown in Figure 9, in the multi-tenant container orchestration system shown in Figure 10, in the server-side container orchestration cluster, the first untrusted control plane node may further include a synchronizer, and the untrusted API server communicates with the trusted admission controller in the tenant trusted container orchestration cluster via the synchronizer.

[0112] In addition, the tenant container orchestration cluster may also include a tenant untrusted container orchestration cluster. The tenant untrusted container orchestration cluster is located between the tenant trusted container orchestration cluster and the server container orchestration cluster. The tenant untrusted container orchestration cluster includes a second untrusted control plane node and a second untrusted data plane node. The second untrusted control plane node includes a second untrusted API server, and the second untrusted data plane node includes a second untrusted Kubelet.

[0113] The trusted admission controller in the tenant's trusted container orchestration cluster and the first untrusted API server communicate via the second untrusted API server, and the untrusted container runtime manager CRI Shim in the server-side container orchestration cluster sends the container group specification of the scheduled first trusted container group to the trusted Kubelet via the second untrusted Kubelet.

[0114] As described above with reference to Figures 1 to 10, a hierarchical container orchestration system, a container orchestration method, and a multi-tenant container orchestration system according to embodiments of this specification are described. The hierarchical container orchestration system can be implemented using hardware, software, or a combination of hardware and software.

[0115] Figure 11 shows an example schematic diagram of a hierarchical container orchestration system 1100 implemented on a computer system according to an embodiment of the present specification. As shown in Figure 11, hierarchical container orchestration system 1100 may include at least one processor 1110, a memory (e.g., non-volatile memory) 1120, a storage 1130, and a communication interface 1140. The at least one processor 1110, the storage 1120, the storage 1130, and the communication interface 1140 are connected together via a bus 1160. The at least one processor 1110 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the aforementioned element implemented in software form).

[0116] In one embodiment, computer executable instructions are stored in a memory, which, when executed, cause at least one processor 1110 to: in response to a trusted API server in a trusted container orchestration cluster receiving an API object deployment request, convert the API object deployment request into the requested API object via the trusted API server and generate a corresponding first trusted container group, and store the first trusted container group in a trusted data storage system in the trusted container orchestration cluster; in response to a trusted admission controller in the trusted container orchestration cluster intercepting the first trusted container group, mask field information of the container group configuration information of the first trusted container group via the trusted admission controller to obtain the first trusted container group. The method further comprises the steps of: creating a shadow container group for the untrusted container group, forwarding the shadow container group to the untrusted API server for container group scheduling via the scheduler; after the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, forwarding the container group specification of the scheduled shadow container group to the trusted container group manager via the untrusted container runtime manager, where the container group specification includes container group configuration information and resource allocation information; and pulling up the first trusted container group from the trusted API server according to the container group configuration information of the shadow container group, and running the first trusted container group in the trusted container runtime using the resources allocated based on the received resource allocation information.

[0117] It should be understood that, when executed, the computer-executable instructions stored in the memory enable the at least one processor 1110 to perform the various operations and functions described above in conjunction with FIG. 1 to FIG. 10 in the various embodiments of this specification.

[0118] According to one embodiment, a program product such as a machine-readable medium (e.g., a non-transitory machine-readable medium) is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software form) that, when executed by a machine, cause the machine to perform the various operations and functions described above in conjunction with Figures 1-10 in the various embodiments of this specification. Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer or processor of the system or device is caused to read and execute the instructions stored in the readable storage medium.

[0119] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of the present invention.

[0120] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0121] According to one embodiment, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the processor performs the various operations and functions described above in conjunction with Figures 1 to 10 in various embodiments of this specification.

[0122] Those skilled in the art will appreciate that the various embodiments disclosed above may be modified and altered in various ways without departing from the essence of the invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims.

[0123] It should be noted that not all steps and units in the above processes and system structure diagrams are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0124] In the above embodiments, hardware unit or module can be realized by mechanical means or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. Hardware unit or processor can also include programmable logic or circuit (such as general purpose processor or other programmable processor), can be temporarily set up to complete the corresponding operation by software. Concrete implementation (mechanical means or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0125] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0126] The foregoing description of the present disclosure is provided to enable any person skilled in the art to implement or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is intended to be consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A hierarchical container orchestration system, comprising: A trusted container orchestration cluster, including trusted control plane nodes and trusted data plane nodes, wherein the trusted control plane nodes include a trusted API server, a trusted admission controller, and a trusted data storage system, and the trusted data plane nodes include a trusted container group manager, a trusted container runtime manager, and a trusted container runtime; And An untrusted container orchestration cluster, including untrusted control plane nodes and untrusted data plane nodes, wherein the untrusted control plane nodes include an untrusted API server and a container group scheduler, and the untrusted data plane nodes include an untrusted container group manager, an untrusted container runtime manager, and an untrusted container runtime, Wherein, after intercepting the first trusted container group generated by the trusted API server in response to the received API object deployment request, the trusted admission controller performs field information masking on the container group configuration information of the first trusted container group to obtain the shadow container group of the first trusted container group, and forwards the shadow container group to the untrusted API server for container group scheduling via the scheduler, After the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, the untrusted container runtime manager forwards the container group specification of the scheduled shadow container group to the trusted container group manager, and the container group specification includes container group configuration information and resource allocation information, The trusted container group manager pulls up the first trusted container group from the trusted API server according to the received container group configuration information, and runs the first trusted container group in the trusted container runtime using the resources allocated based on the received resource allocation information.

2. The hierarchical container orchestration system according to claim 1, wherein, The trusted admission controller performing field information masking on the container group configuration information of the first trusted container group to obtain the shadow container group of the first trusted container group includes: The trusted admission controller performs field information masking on the container group configuration information of the first trusted container group based on a privacy protection policy to obtain the shadow container group of the first trusted container group.

3. The hierarchical container orchestration system according to claim 2, wherein, The trusted admission controller performing field information masking on the container group configuration information of the first trusted container group based on a privacy protection policy to obtain the shadow container group of the first trusted container group includes: For each field in the container configuration information of the first trusted container group, the trusted admission controller: Determines whether the field is a privacy field; In response to determining that the field is a privacy field, determines whether removing the field will affect the untrusted execution in the untrusted container orchestration cluster; If it is determined that it does not affect the untrusted execution in the untrusted container orchestration cluster, the field is removed from the container group configuration information for field information masking, If it is determined that it affects the untrusted execution in the untrusted container orchestration cluster, the field information of the field is replaced with insensitive data for field information masking.

4. The hierarchical container orchestration system according to claim 3, wherein, The trustworthy admission controller further includes: masking field information of the container group configuration information of the first trustworthy container group to obtain the shadow container group of the first trustworthy container group; If it is determined that the field is not a privacy field, determine whether the field requires other masked API objects; If no other masked API objects are required, retain the field; If other masked API objects are required, determine whether retaining or removing the field will trigger an execution error in the untrustworthy container orchestration cluster and / or the trustworthy container orchestration cluster; If it is determined that removing the field will trigger an execution error in the trustworthy container orchestration cluster and retaining the field will not trigger an execution error in the untrustworthy container orchestration cluster, retain or modify the field information of the field; If it is determined that removing the field will not trigger an execution error in the trustworthy container orchestration cluster, and / or retaining the field will trigger an execution error in the untrustworthy container orchestration cluster, remove the field from the container configuration information.

5. The hierarchical container orchestration system according to claim 1, wherein, The trustworthy control plane node further includes a trustworthy customization controller. The trustworthy container group manager sends the container group configuration information of the shadow container group to the trustworthy API server, so that the trustworthy customization controller performs container group integrity verification based on the container group configuration information of the shadow container group and the container group configuration information of the first trustworthy container group stored in the trustworthy data storage system. After the container group integrity verification is successful, the trustworthy container group manager pulls up the first trustworthy container group from the trustworthy API server according to the received container group configuration information.

6. The hierarchical container orchestration system according to claim 1, wherein, The trustworthy control plane node further includes a trustworthy customization controller, and the trustworthy admission controller creates a container group template for indicating scheduling constraints and persists the container group template to the trustworthy data storage system. The untrustworthy container runtime manager further forwards the scheduling information of the first trustworthy container group to the trustworthy container group manager, and the trustworthy container group manager sends the scheduling information to the trustworthy API server to perform scheduling constraint verification according to the scheduling constraints through the trustworthy customization controller. After the scheduling constraint verification is successful, the trustworthy container group manager pulls up the first trustworthy container group from the trustworthy API server according to the received container group configuration information and starts the first trustworthy container group in the trustworthy container runtime using the resources allocated based on the received resource allocation information.

7. The hierarchical container orchestration system according to claim 1, wherein, Before the trustworthy container group manager pulls up the first trustworthy container group from the trustworthy API server, create an indefinite lease for the first trustworthy container group.

8. The hierarchical container orchestration system according to claim 1, wherein After receiving the API object deployment request, the API server converts the API object deployment request into an API object and generates a corresponding first trusted container group. In response to the API object having data sensitivity, the API object and the first trusted container group are stored in the trusted data storage system. Or, in response to the API object not having data sensitivity, the API object is encrypted and stored in the untrusted data storage system, and the first trusted container group is stored in the trusted data storage system.

9. The hierarchical container orchestration system according to claim 1, wherein, In response to receiving a first container group deletion request for deleting the first trusted container group, the trusted admission controller sends a second container group deletion request to the untrusted API server, and the trusted container group manager stops running the first trusted container group. In response to the first trusted container group stopping running, the trusted API server deletes the first trusted container group. In response to receiving the second container group deletion request, the untrusted API server deletes the shadow container group. And in response to the shadow container group being deleted, the untrusted container group manager reclaims the resources allocated to the shadow container group.

10. A multi-tenant container orchestration system, comprising: At least two tenant container orchestration clusters, each tenant container orchestration cluster includes a tenant trusted container orchestration cluster. The tenant trusted container orchestration cluster includes a trusted control plane node and a trusted data plane node. The trusted control plane node includes a trusted API server, a trusted admission controller, and a trusted data storage system. The trusted data plane node includes a trusted container group manager, a trusted container runtime manager, and a trusted container runtime. And A server-side container orchestration cluster, including a first untrusted control plane node and a first untrusted data plane node. The first untrusted control plane node includes a first untrusted API server and a container group scheduler. The untrusted data plane node includes a first untrusted container group manager, an untrusted container runtime manager, and an untrusted container runtime. Wherein, after intercepting the first trusted container group generated by the trusted API server in response to an API object deployment request initiated by a tenant via a secure communication channel, the trusted admission controller masks the field information of the container group configuration information of the first trusted container group to obtain a shadow container group of the first trusted container group, and forwards the shadow container group to the untrusted API server for container group scheduling via the scheduler. After the untrusted container group manager pulls up the shadow container group from the untrusted API server and allocates resources, the untrusted container runtime manager forwards the container group specification of the scheduled shadow container group to the trusted container group manager. The container group specification includes container group configuration information and resource allocation information. The trusted container group manager pulls the first trusted container group from the trusted API server according to the container group configuration information of the shadow container group, and runs the first trusted container group in the trusted container runtime using the resources allocated based on the received resource allocation information.

11. The multi-tenant container orchestration system according to claim 10, wherein, The first untrusted control plane node further includes a synchronizer, and the untrusted API server communicates with the trusted admission controller via the synchronizer.

12. The multi-tenant container orchestration system according to claim 10, wherein, The tenant container orchestration cluster further includes a tenant untrusted container orchestration cluster located between the tenant trusted container orchestration cluster and the server-side container orchestration cluster. The tenant untrusted container orchestration cluster includes a second untrusted control plane node and a second untrusted data plane node. The second untrusted control plane node includes a second untrusted API server, and the second untrusted data plane node includes a second untrusted container group manager. The trusted admission controller and the first untrusted API server communicate via the second untrusted API server, and the untrusted container runtime manager sends the container group specification and resource allocation information of the scheduled first trusted container group to the trusted container group manager via the second untrusted container group manager.

13. A container orchestration method, comprising: In response to a trusted API server in a trusted container orchestration cluster receiving an API object deployment request, converting the API object deployment request into a requested API object via the trusted API server and generating a corresponding first trusted container group, and storing the first trusted container group in a trusted data storage system in the trusted container orchestration cluster; In response to the trusted admission controller in the trusted container orchestration cluster intercepting the first trusted container group, masking the field information of the container group configuration information of the first trusted container group via the trusted admission controller to obtain a shadow container group of the first trusted container group, and forwarding the shadow container group to an untrusted API server for container group scheduling via a scheduler; After the untrusted container group manager pulls the shadow container group from the untrusted API server and allocates resources, forwarding the container group specification of the scheduled shadow container group to the trusted container group manager via the untrusted container runtime manager, where the container group specification includes container group configuration information and resource allocation information; Pulling the first trusted container group from the trusted API server by the trusted container group manager according to the container group configuration information of the shadow container group, and running the first trusted container group in the trusted container runtime using the resources allocated based on the received resource allocation information.

14. The container orchestration method according to claim 13, further comprising: In response to receiving a first container group deletion request for deleting the first trusted container group, sending a second container group deletion request to the untrusted API server via the trusted admission controller, and stopping running the first trusted container group via the trusted container group manager. In response to the first trusted container group stopping running, delete the first trusted container group via the trusted API server; In response to receiving the second container group deletion request, delete the shadow container group via the untrusted API server, and in response to the shadow container group being deleted, perform resource reclamation for the resources allocated to the shadow container group via the untrusted container group manager.

15. A container orchestration system, comprising: At least one processor; A memory coupled to the at least one processor; And A computer program stored in the memory, the at least one processor executing the computer program to implement the container orchestration method as claimed in claim 13 or 14.

16. A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the container orchestration method as claimed in claim 13 or 14.

17. A computer program product comprising computer program instructions, the computer program instructions being executed by a processor to implement the container orchestration method as claimed in claim 13 or 14.

Images