A method, computer system, and computer program for modifying isolation of containerized workloads.

The method dynamically migrates containerized workloads from cgroup-based containers to virtual machines in response to threats or compliance changes, addressing security vulnerabilities and resource inefficiencies in container-based virtualization.

JP7580397B2Active Publication Date: 2024-11-11INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021566234
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-05-09
Filing Date
2020-05-04
Publication Date
2024-11-11
Estimated Expiration
2040-05-04

AI Technical Summary

Technical Problem

Existing container-based virtualization technologies are vulnerable to host kernel vulnerabilities, leading to potential security breaches and inefficiencies due to the need for resource-intensive virtual machines to isolate containers.

Method used

A method and system for dynamically changing the isolation of containerized workloads by migrating them from cgroup-based containers to virtual machines in response to triggering factors like threats or compliance changes, using checkpoint/restore in userspace to maintain service continuity.

Benefits of technology

Enhances security and resource efficiency by isolating containers effectively from kernel vulnerabilities without service interruption, adapting to changing threat levels and compliance requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580397000001
    Figure 0007580397000001
  • Figure 0007580397000002
    Figure 0007580397000002
  • Figure 0007580397000003
    Figure 0007580397000003
Patent Text Reader

Abstract

Methods, apparatus, and computer program products are provided for dynamically changing the isolation of a containerized workload in response to the detection of a triggering factor. In some embodiments, the workload is containerized using a default container runtime (e.g., runC), which spawns one or more cgroup-based containers on a compute node using resource-limiting capabilities, including cgroups and namespaces, of the compute node's host kernel. In some embodiments, at least a portion of the containerized workload is migrated from execution in one or more cgroup-based containers to one or more virtual machines (VMs) launched by a standby container runtime (e.g., runV) in response to a triggering factor, such as a host kernel vulnerability. In some embodiments, CRIU, or user-space checkpoint / restore, is used to live-migrate the cgroups and namespaces of one or more cgroup-based containers to one or more VM runtimes on one or more VMs without service interruption.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to the field of information processing, and more particularly to dynamically changing isolation of containerized workloads in response to detection of a triggering factor. [Background technology]

[0002] An emerging information technology (IT) delivery model is cloud computing, where shared resources, software, and information are delivered on demand to computers and other devices over the Internet. Cloud computing can significantly reduce IT costs and complexity while improving workload optimization and service delivery. In this approach, application instances can be hosted and made available from Internet-based resources accessible through a traditional Web browser over HTTP. An example application might provide a common set of messaging capabilities, such as email, calendar, contact management, and instant messaging. Users would then access the service directly over the Internet. Using this service, an enterprise would place its email, calendar, and / or collaboration infrastructure in the cloud, and end users would use an appropriate client to access their email or perform calendar operations.

[0003] Cloud compute resources are typically housed in large server farms running one or more network applications, typically using a virtualization architecture where applications run inside virtual servers, or so-called "virtual machines" (VMs), that are mapped to physical servers in a data center facility. Virtual machines typically run on top of a hypervisor, a control program that allocates physical resources to virtual machines. Modern hypervisors often use hardware-assisted virtualization, which provides efficient and complete virtualization primarily by using virtualization-specific hardware capabilities of the host CPU.

[0004] Operating system (OS)-level virtualization is another approach to virtualization. OS-level virtualization allows the operating system kernel to partition computer resources by supporting multiple isolated user space instances, which are usually called containers. Therefore, this approach to virtualization is often called container-based virtualization. More commonly, OS-level virtualization includes, but is not limited to, "Container" on Linux(R), "Jail" on FreeBSD, and "Zone" on Solaris. Containers can be implemented by, for example, Linux(R) Container (LXC), Docker, and CoreOS Rocket (rkt). Containers may be indistinguishable from individual machines to end users. Container-based virtualization (e.g., Docker) is widely used in cloud environments. For example, in many current data centers, workloads run inside containers. Containers can provide better agility and orchestration in response to changing demands from workloads. A number of technologies are used in container-based virtualization, including, by way of example, namespaces and cgroups, as described below. The registered trademark Linux® is used pursuant to sublicense from the Linux Foundation, the exclusive licensee of Linus Torvalds, the worldwide owner of such trademark.

[0005] In a typical container cloud computing environment, hosts run an operating system, such as the Linux® kernel. The term "container" mentioned above refers to an OS-level virtualization mechanism that uses a single operating system kernel to run isolated computing workloads (containers) on a controlling host. This technique effectively partitions the resources managed by a single operating system into isolated groups and better balances competing demands for resource usage among the isolated groups. In contrast to other types of virtualization, neither instruction-level emulation nor just-in-time compilation is required. Furthermore, containers can execute instructions that are native to the core CPU without special interpretation mechanisms. By providing a way to create and enter containers, the operating system creates the illusion to each application that it is running on a separate machine, while at the same time each application shares many of the underlying resources.

[0006] The Linux(R) kernel has a feature called "namespaces". Linux(R) kernel namespaces, a main component of Linux(R) containers, isolate applications into various "user spaces" such as network, process, user, and file system. Namespaces isolate and virtualize system resources for a collection of processes. Some examples of resources that can be virtualized include process IDs, hostnames, user IDs, and the like. A namespace typically refers to a type of namespace as well as a specific instance of that type. The Linux(R) operating system is initialized with a single instance of each type of namespace. After initialization, additional namespaces can be created or attached.

[0007] The Linux(R) kernel also provides functionality called Control Groups, also known as "cgroups", that allows for the limiting and prioritization of resources (CPU, memory, block I / O, network, etc.). The cgroup functionality limits various host resources such as CPU number and usage, disk performance, memory, and other process limits.

[0008] Container technology combines kernel support for cgroups and namespaces to provide isolated execution environments for applications.

[0009] Containers share the kernel with the host operating system. Therefore, containers can be used to exploit various kernel vulnerabilities. If the host kernel is configured poorly, the host kernel vulnerabilities can introduce security holes through which containers can gain non-privileged access to other containers or the host system. A good example of such a vulnerability is the kernel local privilege escalation "Dirty COW" (CVE-2016-5195). There are known exploits that allow malicious users to "escape" containers and gain access to the kernel and other containers. No doubt additional host kernel vulnerabilities and associated exploits will continue to be discovered.

[0010] Solutions addressing this issue have so far been inadequate. One such approach involves isolating container execution using one or more virtual machines. In this approach, container workloads are launched inside one or more virtual machines instead of the usual cgroup-based containers. However, virtual machines are typically slower and more resource intensive than cgroup-based containers. As a result, this approach (i.e., always running workloads inside one or more virtual machines) leads to a severe waste of resources.

[0011] Therefore, there is a need in the art to address the aforementioned problems. Summary of the Invention

[0012] Viewed from a first aspect, the present invention provides a method for modifying isolation of containerized workloads in a system having one or more compute nodes each having a host kernel, the method comprising: containerizing the workload using a default container runtime, where the default container runtime uses resource constraining capabilities, including cgroups and namespaces, of the compute node's host kernel to generate one or more cgroup-based containers on the compute node; migrating at least a portion of the containerized workloads running in the one or more cgroup-based containers generated by the default container runtime to one or more virtual machines (VMs) launched by a standby container runtime in response to detection of a triggering factor, where the triggering factor is selected from the group consisting of a perceived threat, a change in compliance requirements, and combinations thereof.

[0013] Viewed from a further aspect, the invention provides a computer system for modifying isolation of a containerized workload, the computer system including one or more processors, one or more computer readable storage devices, and program instructions stored in the at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, the program instructions being executable to: containerize the workload using a default container runtime, the default container runtime using resource limiting capabilities, including cgroups and namespaces, of a host kernel of the compute node to generate one or more cgroup-based containers on the compute node; and migrate at least a portion of the containerized workload running in the one or more cgroup-based containers generated by the default container runtime to one or more virtual machines (VMs) launched by a standby container runtime in response to detection of a triggering factor, the triggering factor being selected from the group consisting of a perceived threat, a change in compliance requirements, and combinations thereof.

[0014] Viewed from a further aspect, the present invention provides a computer program product for modifying isolation of containerized workloads in a system having one or more compute nodes, each having a host kernel, the computer program product including a computer readable storage medium storing instructions readable by a processing circuit for execution by the processing circuit to perform a method for performing the steps of the present invention.

[0015] Viewed from a further aspect, the present invention provides a computer program stored on a computer readable medium and loadable into an internal memory of a digital computer, the computer program comprising software code portions for performing each of the steps of the present invention when said program is executed on a computer.

[0016] Embodiments of the present disclosure include methods, apparatus, and computer program products for dynamically changing isolation of a containerized workload in response to detection of a triggering factor, such as a perceived threat or a change in compliance requirements, or both. By way of example, isolation of a containerized workload may be increased to dynamically respond to changes in the threat level of a host operating system that executes the containerized workload. In some embodiments, a workload is containerized using a default container runtime (e.g., runC), which spawns one or more cgroup-based containers on a compute node using resource-limiting capabilities, including cgroups and namespaces, of a host kernel of the compute node. In some embodiments, at least a portion of a containerized workload is migrated from running in one or more cgroup-based containers to one or more virtual machines (VMs) launched by a standby container runtime (e.g., runV) in response to a perceived threat, such as a vulnerability in the host kernel. In some embodiments, using CRIU, or checkpoint / restore in userspace, cgroups and namespaces of one or more cgroup-based containers are live migrated to one or more VM runtimes on one or more VMs without service interruption.

[0017] The present invention will now be described, by way of example only, with reference to preferred embodiments illustrated in the following drawings.

[0018] The embodiments will now be described with reference to the accompanying drawings, in which like designations represent like components. [Brief description of the drawings]

[0019] [Figure 1]1 illustrates a cloud computing node according to one or more embodiments. [Diagram 2] 1 illustrates a cloud computing environment in accordance with one or more embodiments. [Diagram 3] 1 illustrates abstraction model layers in accordance with one or more embodiments. [Figure 4] 1 illustrates a plurality of compute nodes, at least one of which includes a plurality of virtual machines, according to one or more embodiments. [Diagram 5] 1 illustrates a plurality of compute nodes, at least one of which includes a plurality of containers, according to one or more embodiments. [Figure 6] 1 illustrates a container orchestration system including multiple compute nodes, at least one of which includes a container runtime (e.g., runC) that launches multiple containers, according to one or more embodiments. [Figure 7] A container orchestration system according to one or more embodiments is depicted that includes a plurality of compute nodes, where at least one of the compute nodes includes a container runtime (e.g., runV) that has been modified to launch workloads in a plurality of traditional virtual machines rather than containers created using cgroups and namespaces. [Figure 8] In accordance with one or more embodiments, a container orchestration system is illustrated that includes multiple compute nodes, at least one of which includes a hypervisor-based container runtime (e.g., runV) capable of launching multiple running containers and multiple virtual machines, and a checkpoint / restore in user space (CRIU) that is utilized in response to detection of a triggering factor to live migrate cgroups and namespaces of the running containers from the host to the multiple virtual machines. [Figure 9]FIG. 1 is a flow diagram of an illustrative method for dynamically changing isolation of containerized workloads in response to detection of a triggering factor by live migrating cgroups and namespaces of multiple running containers from a host to multiple virtual machines using CRIU, in accordance with one or more embodiments. [Figure 10] 10 illustrates a container orchestration system corresponding to that of FIG. 8 but further employing traffic shaping via a service mesh, in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] According to one or more embodiments, isolation of a containerized workload is dynamically altered based on detection of a triggering factor, such as a perceived threat or a change in compliance requirements, or both. By way of example, isolation of a containerized workload may be increased to dynamically respond to changes in the threat level of a host operating system that executes the containerized workload. In one or more embodiments, a workload is containerized using a default container runtime (e.g., runC), which spawns one or more cgroup-based containers on a compute node using resource-limiting capabilities, including cgroups and namespaces, of the compute node's host kernel. In one or more embodiments, at least a portion of the containerized workload is migrated from running in the one or more cgroup-based containers to one or more virtual machines (VMs) launched by a standby container runtime (e.g., runV) in response to a perceived threat (e.g., a vulnerability in the host kernel) or other triggering factor (e.g., a change in compliance requirements). In some embodiments, using CRIU, or user space checkpoint / restore, cgroups and namespaces of one or more cgroup-based containers are live migrated to one or more VM runtimes on one or more VMs without service interruption.

[0021] It should be appreciated that although this disclosure includes detailed descriptions of cloud computing, implementation of the teachings described herein is not limited to cloud computing environments, rather, embodiments of the invention may be implemented in connection with any other type of computing environment now known or later developed.

[0022] Cloud computing is a service delivery model that enables on-demand, convenient network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0023] Its features are as follows: On-Demand Self-Service: Cloud consumers can automatically and unilaterally provision computing capacity, such as server time and network storage, as needed, without the need for human interaction with the provider of the service. Pervasive Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client and thick-client platforms (e.g., mobile phones, laptops, and personal digital assistants (PDAs)). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Generally, there is a sense of location independence in that consumers have no control or knowledge of the exact location of the resources provided to them, although at higher levels of abstraction they may be able to specify a location (e.g., country, state, or data center). Rapid Elasticity: Capacity is rapidly and elastically provisioned, in some cases automatically, to quickly scale out and quickly release to quickly scale in. In many cases, to the consumer, capacity available for provisioning appears unlimited, and any amount can be purchased at any time. Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0024] The service model is as follows: Software as a Service (SaaS): The ability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g. web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the capabilities of the individual applications, with the possible exception of limited per-user application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The capability provided to a consumer is the provisioning of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases limited control over selected networking components (e.g., host firewalls). Containers as a Service (CaaS) is a cloud computing service model that enables users to deploy and manage containers, applications, and clusters through container-based virtualization. CaaS generally sits on the spectrum of cloud computing service models between Infrastructure as a Service (IaaS) and Platform as a Service (PaaS), both of which are discussed above. However, CaaS is typically considered a subset of IaaS, although CaaS includes containers as its fundamental resources (as opposed to the virtual machines or bare-metal hardware host systems traditionally used to support IaaS environments).

[0025] The deployment models are as follows: Private Cloud: The cloud infrastructure is operated solely for an organization. The cloud infrastructure can be managed by the organization or a third party and can be on-premise or off-premise. Community Cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance awareness). The cloud infrastructure can be managed by the organization or a third party and can be on-premise or off-premise. Public Cloud: Cloud infrastructure is made available to the public or large industry organizations and is owned by an organization that sells cloud services. Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) where each cloud remains a unique entity but is bound by standardized or proprietary technologies that enable data and application portability (e.g. cloud bursting for load balancing between clouds).

[0026] Cloud computing environments are service-oriented and focus on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0027] Referring now to Figure 1, a schematic diagram of an example cloud computing node is shown. Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to be denoted in any way.

[0028] Reference is now made to FIG. 1. A schematic diagram of an example cloud computing node is shown. Cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to imply any limitation as to the scope of use or functionality of the embodiments of the present invention described herein. In any event, cloud computing node 10 may implement and / or perform any of the functionality described above.

[0029] Within cloud computing node 10 is computer system / server 12, which is operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or any combination thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems or devices, and the like.

[0030] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may also be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0031] 1, computer system / server 12 in cloud computing node 10 is shown in the form of a general purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0032] Bus 18 may represent any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor bus or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus.

[0033] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, and removable and non-removable media.

[0034] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown and typically referred to as a "hard drive") and other non-removable non-volatile media (e.g., a "solid state drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and / or writing to a removable non-volatile optical disk, such as a CD-ROM (compact disc read-only memory), DVD-ROM, or other optical medium. In such instances, each may be connected to the bus 18 by one or more data media interfaces. As described further below, memory 28 may include a computer program product that stores a set (e.g., at least one) of program modules 42 that include computer-readable instructions configured to perform one or more features of the present invention.

[0035] A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. In some embodiments, the program modules 42 are adapted to generally perform one or more functions and / or methodologies of one or more embodiments of the present invention.

[0036] The computer system / server 12 may further communicate with one or more external devices 14, such as, for example, a keyboard, a pointing device, a display 24, or one or more devices that allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), or any combination thereof. Such communication may occur through an input / output (I / O) interface 22. The computer system / server 12 may further communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or any combination thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 through a bus 18. Although not shown, it will be appreciated that other hardware and / or software components could be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0037] Referring now to FIG. 2, an illustrative cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or any combination thereof, can communicate. The nodes 10 may communicate with each other. The nodes 10 may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or any combination thereof, as described above (not shown). This allows the cloud computing environment 50 to provide infrastructure, platform, or software, or any combination thereof, as a service, for which the cloud consumer does not need to maintain resources on a local computing device. Of course, the types of computing devices 54A-N shown in FIG. 2 are for illustrative purposes only, and the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0038] Referring now to FIG. 3, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 2) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 3 are for illustrative purposes only, and the present invention is not limited thereto. As shown, the following layers and corresponding functions are provided:

[0039] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (Reduced Instruction Set Computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0040] The virtualization layer 70 provides an abstraction layer from which virtual entities such as virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 may be provided.

[0041] In one example, management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides tracking of costs as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of such resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and achievement 85 provides advance provisioning and procurement of cloud computing resources in anticipation of future demand according to SLAs.

[0042] The workload tier 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and mobile desktop 96.

[0043] 4 and 5 respectively show a number of compute nodes 400 and 500 (also referred to as "host devices") that may be used in accordance with some embodiments of the present invention. Compute node 400 of FIG. 4 includes a number of exemplary system VMs, i.e., fully virtualized VMs, that completely replace the target real machine and provide the level of functionality required to run a complete operating system 403. Compute node 500 of FIG. 5 includes a number of exemplary OS-level virtualization systems that enable partitioning of computer resources by kernel support for isolated user space instances, typically called containers, that may have the appearance and feel of a real machine to an end user. Some embodiments of the present invention may be used with various types of virtualization. By way of example, some embodiments of the present invention may be used with virtual machine management (e.g., OpenStack) and container management (e.g., Kubernetes).

[0044] As an example, a container management system (e.g., Kubernetes, Docker Swarm) may be utilized to manage the lifecycle (Create, Read, Update, and Delete (CRUD)) of containers in a cluster-wide system. As a typical example, when a container creation request is received, a scheduler selects a host on which the requested container will run. An agent in the selected host then launches the container. It should be understood that the terms "host" and "node" are used interchangeably herein and refer to a hardware device or system with at least a processor, memory, and a communication mechanism for interacting with other hosts / nodes.

[0045] 4 and 5 show compute nodes 400 and 500 using full virtualization and OS-level virtualization, respectively. Some embodiments of the invention may be used with any of these types of compute nodes, and may also be used in hybrid environments with a combination of these compute nodes across single or multiple compute nodes.

[0046] As shown in FIG. 4, each compute node 400 includes hardware 406, which may include a processor (or CPU) 407, memory 408, a network interface card (NIC) 409, and a disk drive 410. The disk drive 410 may include a solid state drive or a hard disk drive, or some combination of the two. The compute nodes 400 run a host operating system 405 on the hardware. The compute nodes 400 further include a hypervisor 404 for sharing and managing the hardware 406, allowing multiple different environments 401, isolated from one another, to run on the same physical machine 400. The hypervisor 404 may use hardware-assisted virtualization, which provides efficient and complete virtualization by primarily using virtualization-specific hardware capabilities of the host CPU 407. Each compute node 400 includes one or more virtual machines 401, each of which includes a guest operating system 403 and one or more application programs (or applications) 402 that run on the guest operating system 403.

[0047] Similarly, as shown in Figure 5, each compute node 500 includes hardware 506 that may include a processor (or CPU) 507, memory 508, a network interface card (NIC) 509, and a disk drive 510. The disk drive 510 may include a solid state drive or a hard disk drive, or some combination of the two. The compute nodes 500 run a host operating system 505 on the hardware. Each compute node 500 includes one or more containers 501, each of which includes one or more applications 502.

[0048] According to some embodiments, a compute node 500 may include one or more pods 503, each of which includes one or more containers 501, each of which includes one or more applications 502. By way of example, in Kubernetes, containers run within pods.

[0049] "Kubernetes" is a portable, extensible, open-source platform for managing containerized workloads and services. Kubernetes facilitates both declarative configuration and automation. The Kubernetes project was open-sourced by Google in 2014. Kubernetes orchestrates compute, networking, and storage infrastructure for user workloads. Kubernetes is an example of an orchestration framework. Other orchestration frameworks include, but are not limited to, Docker Swarm, LXD, Rancher, and Apache Aurora / Mesos.

[0050] Containerized workloads across multiple compute nodes may be managed by a container orchestration manager (COM). An example of a container orchestration manager (COM) is the Kubernetes master.

[0051] Several binary components (e.g., master components, node components, and addons) are utilized to provide a functioning Kubernetes cluster.

[0052] Master components provide the control plane for a Kubernetes cluster (also called the "Kubernetes control plane"). Master components can include, but are not limited to, kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager. Master components make global decisions about the Kubernetes cluster. For example, master components handle scheduling. Additionally, master components are used to detect and respond to cluster events. For example, master components are responsible for starting new pods when the "replicas" field of a replication controller is not satisfied. Master components can run on any machine in the cluster. However, setup scripts typically start all master components on the same machine and do not run user containers on that machine.

[0053] Node components run on every compute node in a Kubernetes cluster. Node components are responsible for maintaining running pods and providing the Kubernetes runtime environment. Node components may include, but are not limited to, the kubelet, kube-proxy, and the container runtime.

[0054] The kubelet is an agent that ensures that containers are running within a Pod. The kubelet ensures that the containers specified in a set of PodSpecs, provided through various mechanisms, are running and healthy.

[0055] kube-proxy is a network proxy that enables abstraction of Kubernetes services by maintaining network rules and performing connection forwarding on compute nodes.

[0056] A container runtime is software responsible for running containers. More specifically, a container runtime is a node component that handles the lifecycle of containers. A container runtime implements fundamental concepts such as creating, starting, stopping, and removing container workloads. Kubernetes supports several runtimes, including but not limited to Docker, containerd, CRI-O, and rktlet.

[0057] More generally, Kubernetes supports any implementation of the Container Runtime Interface (CRI) that it provides. CRI allows different container runtimes to be easily plugged in. Before the introduction of CRI in Kubernetes 1.5, only the default Docker image repository and its default OCI-compatible runtime, runC, were used. The Open Container Initiative (OCI) created a runtime specification that details the APIs for OCI-compatible container runtimes. runC, runV, and Intel's Clear Containers (also known as "cc-runtime") are examples of OCI-compatible container runtimes. runC has built-in support for CRIU, or user-space checkpoint / restore, which checkpoints and restores containers, as described below. runV is a hypervisor-based Docker runtime for OCI. runV is also known as "Hyper runV".

[0058] CRI runtimes are at a higher level of abstraction and should not be confused with OCI-compatible runtimes. CRI runtimes are also known as "CRI shims". CRI shims include cri-containerd, CRI-O, dockershim, and frakti. Some CRI shims (e.g. cri-containerd, CRI-O, and dockershim) invoke the OCI-compatible runtime, while other CRI shims (e.g. frakti) are monolithic solutions.

[0059] At least some CRI shims support multiple runtimes running on a single compute node. As an example, CRI-O supports the notion of trusted and untrusted sandboxes. In Kubernetes, a combination of one or more VM-based pods and one or more cgroup / namespace-based pods may run on a single compute node, based on pod annotations and the default CRI-O configuration. Containers running inside a VM-based pod may be isolated and managed by namespace and cgroup, similar to how runC is done.

[0060] Addons are pods and services responsible for implementing cluster functionality, including but not limited to cluster DNS (i.e., a DNS server that provides DNS records for Kubernetes services), Dashboards (i.e., a web-based UI for Kubernetes clusters that allows users to manage and troubleshoot applications running in the cluster as well as the cluster itself), Container Resource Monitoring (i.e., responsible for logging general time series metrics about containers in a central database and providing a UI for viewing the data logged in that database), and Cluster Level Logging (i.e., responsible for storing container logs in a central log store with a search / view interface).

[0061] According to one or more embodiments, using the functionality of CRIU, live migration of containerized workloads from one runtime (e.g., runC) to another runtime (e.g., runV) can be achieved without service interruption. Generally, CRIU provides the ability to freeze the state of a running container at the source and restore the container to the same running state at the destination. The CRIU process typically includes the following operations: 1) sync the container's file system between the source and destination (pre-freeze / dump), 2) freeze the container's processes at the source, 3) dump the container at the source (e.g., to a dump file in a checkpoint directory), 4) sync the container's file system between the source and destination (post-freeze / dump), 5) copy the dump file at the destination, 6) resume the container at the destination, 7) resume the container's frozen processes at the destination, 8) stop the container at the source, and 9) destroy the container at the source.

[0062] FIG. 6 illustrates a container orchestration system 600 including multiple compute nodes 602, at least one of which includes a container runtime (e.g., runC) 604 that launches multiple containers 606, according to one or more embodiments. The container orchestration system 600 may also be referred to as a cluster (e.g., the container orchestration system 600 may correspond to a Kubernetes cluster). The compute nodes 602 may be managed by a container orchestration manager (COM) 610. As an example, in Kubernetes, each compute node 602 includes the services (i.e., node components) necessary to run one or more pods 608 and is managed by a Kubernetes master component. The services on each compute node 602 may include a container runtime 604 (e.g., runC), an agent 612 (e.g., kubelet) that listens for instructions from a container orchestration manager (COM) 610 regarding container lifecycle operations executing on that compute node, and a network proxy (e.g., kube-proxy). The container orchestration manager (COM) 610 may include at least a portion of one or more of the Kubernetes master components.

[0063] A container runtime 604 (e.g., runC) manages one or more containers 606 at the compute node level. A container orchestration manager (COM) 610 manages each container runtime 604 (e.g., runC) at the distributed system (Kubernetes cluster) level (i.e., manages one or more container runtimes (e.g., runC)).

[0064] As will be appreciated, containers 606 are spawned on compute nodes 602 using a container runtime 604 (e.g., runC) with resource constraining capabilities, such as cgroups and namespaces, of the compute node's host kernel. runC is a command line tool that spawns and runs containers according to the Open Container Initiative (OCI) specification. runC is a lightweight universal container runtime that provides isolation using Linux® cgroups and namespaces. runC is an example of an OCI-compatible container runtime. As will be appreciated by those skilled in the art, other container runtimes may be used instead of or in addition to runC. As an example, runC may be used with a CRI runtime (CRI shim), such as CRI-O, that calls runC.

[0065] In the embodiment shown in Figure 6, the container orchestration system 600 includes an ingress traffic controller 614. Although the ingress traffic controller 614 is shown separate from the container orchestration manager (COM) 610 in Figure 6, the container orchestration manager (COM) 610 and the ingress traffic controller 614 may run on the same machine. The ingress traffic controller 614 may be configured by an administrator 616.

[0066] In Kubernetes, an ingress is an API object that manages access to services in a Kubernetes cluster from outside the Kubernetes cluster. Access can be configured by creating a collection of rules (called "routing rules") that define which inbound connections can reach which Kubernetes services in the cluster. Traffic routing is controlled by routing rules defined on an ingress resource. Routing rules are typically consolidated in one place (called the "ingress resource"). Ingresses can provide load balancing, SSL termination, and name-based routing, as examples. Ingresses expose HTTP and HTTPS routes from outside the cluster to services in the cluster.

[0067] An ingress controller, such as the ingress traffic controller 614 shown in Figure 6, is responsible for performing the ingress. As an example, Google Compute Engine (GCE) / Google Kubernetes Engine (GKE) deploys an ingress controller on the master. In environments other than GCE / GKE, any of several ingress controllers (e.g., the Nginx ingress controller) are available for deployment.

[0068] FIG. 7 illustrates a container orchestration system 700 including multiple compute nodes 702, according to one or more embodiments, where at least one of the compute nodes includes a container runtime 704 (e.g., runV) that is modified to launch workloads 706 in multiple traditional virtual machines 708 rather than containers created using cgroups and namespaces. This type of setup is useful, by way of example, when the workloads 706 are of a critical nature (e.g., financial transactions and validation of the Hyperledger blockchain) and need to be protected from kernel vulnerabilities such as "Dirty COW". The container orchestration system 700 may also be referred to as a cluster (e.g., the container orchestration system 700 may correspond to a Kubernetes cluster). The compute nodes 702 may be managed by a container orchestration manager (COM) 710. By way of example, in Kubernetes, each compute node 702 includes the services (i.e., node components) necessary to run one or more pods and is managed by a Kubernetes master component. The services on each compute node 702 may include a container runtime 704 (e.g., runV or other hypervisor-based runtime implementation of the OCI runtime specification), an agent 712 (e.g., kubelet) that listens for instructions from a container orchestration manager (COM) 710 regarding container lifecycle operations running on that compute node, and a network proxy (e.g., kube-proxy). The container orchestration manager (COM) 710 may include at least a portion of one or more of the Kubernetes master components.

[0069] A container runtime 704 (e.g., runV) manages workloads 706 at the compute node level. A container orchestration manager (COM) 710 manages each container runtime 704 (e.g., runV) at the distributed system (Kubernetes cluster) level (i.e., manages one or more container runtimes (e.g., runV)).

[0070] Hyper runV (also referred to as "runV") is a hypervisor-based runtime (i.e., an OCI compatible runtime) for the OCI runtime, and functions similarly to runC. However, unlike runC, runV does not use cgroups and namespaces, but instead uses a hypervisor to execute (Docker) images. Those skilled in the art will appreciate that other hypervisor-based runtime implementations of the OCI runtime specification may be used instead of or in addition to runV. Hypervisor-based runtimes such as runV are also referred to herein as "VM-based container runtimes." Other examples of hypervisor-based runtime implementations of the OCI runtime specification include, but are not limited to, Intel(R) Clear Containers (also known as "cc-runtime") and VMWare's vSphere Integrated Containers (VIC). Additionally, runV and other hypervisor-based runtime implementations of the OCI runtime specification may be used with a CRI runtime (CRI shim) that calls into a hypervisor-based runtime, such as CRI-O. Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and / or other countries. VMware is a trademark or registered trademark of VMware, Inc. or its subsidiaries in the United States and / or other jurisdictions.

[0071] 8 illustrates a container orchestration system 800 including multiple compute nodes 802, at least one of which includes a hypervisor-based container runtime 805 (e.g., runV) capable of launching multiple running containers 806 and multiple virtual machines 810, and a checkpoint / restore in user space (CRIU) 820 that is utilized in response to detecting a triggering factor to live migrate cgroups and namespaces of the running containers 806 from the host 802 to the multiple virtual machines 810, according to one or more embodiments. The live migration of cgroups and namespaces from the host 802 to the multiple virtual machines 810 establishes cgroup-based containers 808 running on the multiple virtual machines 810.

[0072] According to one or more embodiments, container orchestration manager (COM) 811 may be configured (e.g., by administrator 801) to respond to detection of a triggering factor by initiating either a full migration mode or a partial migration mode. By way of example, depending on the triggering factor detected, administrator 801 of COM 811 may select a full migration mode that migrates a complete workload from a regular cgroup and namespace-based container runtime to a VM-based container runtime, or a partial migration mode that horizontally scales containers across regular cgroup and namespace-based container runtimes and VM-based container runtimes by leveraging load balancing.

[0073] As an example, a workload running in a cgroup-based container may be migrated in response to the detection of a triggering factor, such as a vulnerability in the host kernel. In some embodiments, this migration may occur without causing an interruption to the service. However, one skilled in the art will appreciate that a workload running in a cgroup-based container may be migrated in response to the detection of one or more other triggering factors instead of or in addition to a vulnerability in the host kernel. This approach may also be used, for example, in scenarios where compliance requirements for a running service need to be changed without interrupting service uptime.

[0074] A container runtime 804 (e.g., runC) manages containers 806 at the compute node level. The container runtime 804 (e.g., runC) is also referred to herein as the “default container runtime.”

[0075] A hypervisor-based container runtime 805 (e.g., runV) manages containers 808 at the compute node level. The hypervisor-based container runtime 805 (e.g., runV) is also referred to herein as a “standby container runtime.”

[0076] The container orchestration manager (COM) 811 manages each container runtime 804 (e.g., runC) and each hypervisor-based container runtime 805 (e.g., runV) at the distributed system (Kubernetes cluster) level (i.e., manages one or more container runtimes 804 (e.g., runC) and one or more hypervisor-based container runtimes (e.g., runV)).

[0077] The checkpoint command checkpoints (i.e., the "C" in CRIU) the current state of the container on the host where the container is currently running. In its default configuration, runC writes checkpoint data to a directory called checkpoint. The checkpoint contains all the information necessary to restore all processes in the container to the same state they had during the checkpoint. This checkpoint data includes open files, memory contents, and file systems. This checkpoint data also includes cgroups and namespaces. According to one or more embodiments, once all the checkpoint data has been written by runC to the checkpoint directory, the container can be restored on a VM launched by runV. The restore command reads the checkpoint data from the checkpoint directory and restores (i.e., the "R" in CRIU) all processes in the container to the same state they had during the checkpoint.

[0078] According to one or more embodiments, checkpointing the runtime state of the container may include temporarily pausing a running container and capturing and storing both the container's in-memory data and file system state to a local disk. In an example implementation, the CRIU may be used to quiesce and dump the container's in-memory data in the form of a set of image files and snapshot the container's file system to capture the container's on-disk state.

[0079] Referring to FIG. 8, a container 806 is instantiated and executed on a host 802. The container 806 includes container memory data and a container file system. Upon checkpointing, data from the container 806 may be stored to a local disk (e.g., disk 510 of FIG. 5). This data includes the container memory data and the container file system. In an exemplary embodiment, the memory data may be stored at / var / lib / container / CONTAINER-ID / statesSTATE-ID / MEM-ID and the file system may be stored at / var / lib / container / CONTAINER-ID / statesSTATE-ID / FS-ID.

[0080] The container orchestration system 800 may also be referred to as a cluster (e.g., the container orchestration system 800 may correspond to a Kubernetes cluster). The compute nodes 802 may be managed by a container orchestration manager (COM) 811. As an example, in Kubernetes, each compute node 802 includes the services (i.e., node components) necessary to run one or more pods 807, 809 and is managed by a Kubernetes master component. The services on each compute node 802 may include a container runtime 804 (e.g., runC), a hypervisor-based container runtime 805 (e.g., runV), an agent 812 (e.g., kubelet) that listens for instructions from the container orchestration manager (COM) 811 regarding container lifecycle operations running on that compute node, and a network proxy (e.g., kube-proxy). The container orchestration manager (COM) 811 may include at least a portion of one or more of the Kubernetes master components.

[0081] In the embodiment shown in FIG. 8, the running containers 806 are cgroup-based containers launched by a container runtime 804 (e.g., runC). Of course, the containers 806 are spawned on the compute nodes 802 using the container runtime 804 (e.g., runC) with resource-constraining capabilities, such as cgroups and namespaces, of the compute node's host kernel. runC is a command-line tool that spawns and runs containers according to the Open Container Initiative (OCI) specification. runC is a lightweight universal container runtime that provides isolation using Linux® cgroups and namespaces.

[0082] runC is an example of an OCI-compatible container runtime. Those skilled in the art will appreciate that other container runtimes may be used instead of or in addition to runC. For example, runC may be used with a CRI runtime (CRI shim), such as CRI-O, which calls runC.

[0083] runV is a hypervisor-based Docker runtime (i.e., OCI-compatible runtime) from Hyper (The Hyper.sh Team) for the OCI runtime, and functions similarly to runC. However, unlike runC, runV does not use cgroups and namespaces, but uses a hypervisor to run (Docker) images. runV is a hypervisor-based Docker runtime, but is hypervisor-agnostic. By way of example, runV can use existing hypervisors such as KVM, Xen, and ESXi. Those skilled in the art will appreciate that other hypervisor-based runtime implementations of the OCI runtime specification may be used instead of or in addition to runV. Other examples of hypervisor-based runtime implementations of the OCI runtime specification include, but are not limited to, Intel's Clear Containers (also known as "cc-runtime") and VMWare's vSphere Integrated Containers (VIC). Additionally, runV and other hypervisor-based runtime implementations of the OCI Runtime Specification may be used in conjunction with a CRI runtime that invokes a hypervisor-based runtime, such as CRI-O (a CRI shim).

[0084] In the embodiment shown in FIG. 8, the container orchestration system 800 includes an ingress traffic controller 814. The ingress traffic controller 814 may be configured by an administrator 816. By way of example, in the embodiment shown in FIG. 8, the administrator 816 may configure the ingress traffic controller 814 to reroute ingress traffic (that was originally destined for a regular cgroup and namespace-based container runtime) to a VM-based container runtime without impacting service uptime.

[0085] 8, the ingress traffic controller 814 is shown separate from the container orchestration manager (COM) 811, but the container orchestration manager (COM) 811 and the ingress traffic controller 814 may run on the same machine. In some embodiments, the container orchestration manager (COM) 811 and the ingress traffic controller 814 may be configured by the same administrator.

[0086] In Kubernetes, an ingress is an API object that manages access to services in a Kubernetes cluster from outside the Kubernetes cluster. Access can be configured by creating a collection of rules (called "routing rules") that define which inbound connections can reach which Kubernetes services in the cluster. Traffic routing is controlled by routing rules defined on an ingress resource. Routing rules are typically consolidated in one place (called the "ingress resource"). Ingresses can provide load balancing, SSL termination, and name-based routing, as examples. Ingresses expose HTTP and HTTPS routes from outside the cluster to services in the cluster.

[0087] An ingress controller, such as the ingress traffic controller 814 shown in Figure 8, is responsible for performing the ingress. As an example, Google Compute Engine (GCE) / Google Kubernetes Engine (GKE) deploys an ingress controller on the master. In environments other than GCE / GKE, any of several ingress controllers (e.g., the Nginx ingress controller) are available for deployment.

[0088] 9 is a flow diagram of an illustrative method 900 for dynamically changing isolation of containerized workloads in response to detection of a triggering factor by live migrating cgroups and namespaces of multiple running containers from a host to multiple virtual machines using CRIU, according to one or more embodiments. Method 900 specifies a preferred ordering of each block, but it should be understood that the various blocks may occur at any point in time relative to one another.

[0089] The method 900 begins by containerizing a workload using a default container runtime (e.g., runC) (block 902). By way of example, the default container runtime may spawn one or more cgroup-based containers on a compute node using resource limiting capabilities, including cgroups and namespaces, of the compute node's host kernel. The default container runtime may be runC or any other suitable default container runtime. Suitable default container runtimes include, but are not limited to, an OCI-compatible container runtime that provides isolation using Linux® cgroups and namespaces. In some embodiments, runC (or any other suitable default container runtime) on a compute node may create one or more cgroup-based containers on the compute node in response to a request from an agent on the compute node that listens for instructions from a container orchestration manager (COM) regarding container lifecycle operations to be performed on the compute node (i.e., the agent sent a request to the default container runtime in response to receiving instructions from the container orchestration manager (COM) that a container be created on the compute node). In other embodiments, the default container runtime may receive requests from an agent indirectly through a CRI runtime (CRI shim), such as CRI-O, that sits between the agent and the default container runtime.

[0090] The method 900 proceeds with detecting a triggering factor (block 904). Illustrative examples of triggering factors include, but are not limited to, a recognition of a threat (e.g., a vulnerability in the host kernel), a change in compliance requirements (e.g., compliance requirements for a running service need to be changed), and combinations thereof. As an example, a container orchestration manager (COM) administrator may learn of a host kernel vulnerability through a monitoring service. In response to the recognized kernel vulnerability, the COM administrator may determine that it is prudent to change the isolation of at least a portion of the containerized workload. As an example, the COM administrator may start a standby container runtime that launches one or more virtual machines to which at least a portion of the containerized workload is then migrated. In some embodiments, in response to the vulnerability, the COM administrator may choose to migrate the complete workload to a VM-based container runtime or choose to horizontally scale containers across regular cgroup and namespace-based container runtimes and VM-based container runtimes by leveraging load balancing (optional block 906, described below).

[0091] Optionally, method 900 proceeds with selecting a full migration mode or a partial migration mode (block 906). Block 906 is illustrated in FIG. 9 using dashed lines to indicate optionality. By way of example, based on one or more characteristics associated with a triggering factor (e.g., the severity of the host kernel vulnerability detected in block 904), a COM administrator may select either a full migration mode (i.e., all of the containerized workloads are migrated to a VM-based container runtime) or a partial migration mode (i.e., the containerized workloads are horizontally scaled across regular cgroup and namespace-based container runtimes and VM-based container runtimes by leveraging load balancing).

[0092] The method 900 then continues by migrating at least a portion of the containerized workloads executing in one or more cgroup-based containers created by the default container runtime to one or more virtual machines launched by the standby container runtime (e.g., runV) in response to detecting the triggering factor at block 904 (and optionally selecting the migration mode at block 906) (block 908). In some embodiments, the COM may automatically send instructions that an "isolate modified container migration" be performed on the compute node in response to detecting the triggering factor.

[0093] In another embodiment, an administrator of the COM may determine whether it is advisable to change the isolation of at least a portion of the containerized workload in response to detecting a triggering factor. If the administrator of the COM determines that it is advisable to change the isolation of at least a portion of the containerized workload, the administrator may cause the COM to send instructions that an “isolation change container migration” be performed on the compute node.

[0094] The “isolate changes container migration” instruction, whether sent automatically or at the direction of a COM administrator, initiates a standby container runtime, such as runV, which starts one or more virtual machines on the compute node, and then initiates a live migration of cgroups and namespaces of at least one of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more virtual machines. The cgroups and namespaces may be live migrated using checkpoint / restore in user space (CRIU), as an example. According to one or more embodiments, the “isolate changes container migration” instruction may include a flag indicating whether a full or partial migration mode was selected in optional block 906.

[0095] The standby container runtime may be runV or any other suitable standby container runtime. Suitable standby container runtimes include, but are not limited to, other hypervisor-based runtime implementations of the OCI Runtime Specification. In some embodiments, runV (or any other suitable standby container runtime) on a compute node may spawn one or more virtual machines on the compute node in response to a request from an agent on the compute node listening for instructions from a container orchestration manager (COM) regarding container lifecycle operations to be performed on the compute node (i.e., the agent sent a request to the standby container runtime in response to receiving instructions from a container orchestration manager (COM) to perform an “isolate-change container migration” on the compute node). In other embodiments, the default container runtime may receive requests from the agent indirectly through a CRI runtime (CRI shim), such as CRI-O, that is interposed between the agent and the default container runtime.

[0096] As described above, according to one or more embodiments, the "isolation change container migration" instruction may include a flag indicating whether a full or partial migration mode was selected in optional block 906. In other words, the migration of a workload from a regular cgroup runtime to a VM-based container runtime may be full (i.e., 100% of the workload is migrated) or partial (i.e., <100% of the workload is migrated). According to one or more embodiments, such dynamic switching of container runtimes based on a perceived threat (or changes in compliance requirements) occurs without interruption of service.

[0097] On the other hand, if the value of the instruction flag indicates that full migration mode was selected in optional block 906, the “isolate change container migration” instruction initiates a standby container runtime that starts one or more virtual machines on the compute node, and then initiates a live migration of the cgroups and namespaces of each of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more virtual machines.

[0098] On the other hand, if the value of the instruction flag indicates that a partial migration mode was selected in optional block 906, the "isolate change container migration" instruction initiates a standby container runtime that starts one or more virtual machines on the compute node, and then initiates a live migration of the cgroups and namespaces of each container of a cgroup-based subset of one or more containers running on the compute node to one or more VM runtimes on the one or more virtual machines. Partial migration may be facilitated by horizontally scaling containers across normal cgroup and VM runtimes, for example.

[0099] The method 900 then continues by rerouting ingress traffic of the migrated containerized workload to one or more VM runtimes (block 910). By way of example, the ingress traffic controller may be configured to reroute ingress traffic (originally destined for the migrated regular cgroup and namespace-based container runtime) to the VM-based container runtime without impacting service uptime. In one or more embodiments where the migration of the workload from the regular cgroup and namespace-based container runtime to the VM-based container runtime is complete (i.e., 100% of the workload has been migrated), the ingress traffic controller may reroute all ingress traffic to the VM-based container runtime. In one or more embodiments where the migration of the workload from the regular cgroup and namespace-based container runtime to the VM-based container runtime is partial (i.e., <100% of the workload has been migrated), the ingress traffic controller may reroute only ingress traffic originally destined for the migrated regular cgroup runtime to the VM-based container runtime.

[0100] Additionally, according to one or more embodiments, the rerouting of ingress traffic for the migrated containerized workloads in block 910 may use traffic shaping via a service mesh. An example of a container orchestration system using traffic shaping via a service mesh is shown in FIG. 10, described below.

[0101] 10 illustrates a container orchestration system 1000 that corresponds to the container orchestration system 800 of FIG. 8, but further employs traffic shaping via a service mesh, in accordance with one or more embodiments. When a container switches runtimes (i.e., the container switches from a regular cgroup and namespace-based container runtime to a VM-based container runtime), runtime metadata about the container (i.e., including but not limited to runtime change events) is sent from one or more service mesh sidecars 1009 (also referred to as "sidecar containers") to a "central controller" (e.g., service mesh controller 1002) to ensure that decisions related to the container runtime can be made. In the embodiment shown in Figure 10, each pod 807 (with a conventional cgroup and namespace-based container runtime) includes a sidecar container 1007, and each pod 809 (with a VM-based container runtime) includes a sidecar container 1009 (the sidecar containers in each pod are shaded in Figure 10 to distinguish them from other containers in the pod). According to one or more embodiments, the service mesh controller 1002 may send routing rule updates to the ingress traffic controller 814 based on the runtime metadata the service mesh controller 1002 receives from the service mesh sidecar 1009.

[0102] There are two logical components required to create a service mesh. The first logical component required to create a service mesh is a pod. A pod is designed to have many containers. The second logical component required to create a service mesh is a container called a "sidecar" (e.g., container sidecar 1009). In a service mesh, each pod contains one or more sidecar containers. A sidecar extends and augments the primary container in the pod. As an example, a sidecar container may monitor things on the primary container and perform one or more tasks on the data resulting from the monitoring operations, thereby alleviating that responsibility of the primary container. In a service mesh, a sidecar may be a service proxy or a data plane.

[0103] A service mesh on Kubernetes may be created using service mesh solutions such as Istio and Linkerd (created by Buoyant), where a "central controller" in the service mesh (e.g., Service Mesh Controller 1002) defines how each sidecar container behaves (except for the sidecar proxy).

[0104] According to one or more embodiments, various traffic shaping policies may be added to the service mesh by setting different runtime priority levels. By way of example, in one or more embodiments, the service mesh controller 1002 may trigger a change in the service mesh routing policy based on whether runtime metadata is received from a regular cgroup and namespace-based container runtime (e.g., runC) or a VM-based container runtime (e.g., runV).

[0105] According to one or more embodiments, a "central controller" in the service mesh (e.g., service mesh controller 1002) shapes traffic based on new metrics (i.e., runtime metadata) currently being fetched from the new container runtime. This may happen dynamically, such as when a service mesh metrics provider is switched at runtime depending on the container runtime engine. As an example, in the embodiment shown in FIG. 10, the service mesh metrics provider is switched at runtime from sidecar container 1007 (with a normal cgroup and namespace based container runtime) to sidecar container 1009 (with a VM-based container runtime). Thus, a "central controller" in the service mesh (e.g., service mesh controller 1002) may include a metrics adaptor 1010 that transparently handles this change.

[0106] According to one or more embodiments, in response to new metrics (i.e., runtime metadata) indicating that the VM-based container runtime is becoming overloaded, the service mesh controller 1002 may generate a service mesh routing policy whereby a certain percentage of ingress traffic is rerouted to another node (i.e., not to the VM-based container runtime).

[0107] 10, according to one or more embodiments, CRIU 820 may be optional for a subset of the blocks (i.e., at least one but not all of the blocks shown in FIG. 10 as CRIU 820). Without a CRIU, container runtime switching may be accomplished by starting a workload on a VM (e.g., VM 810) created by a standby container runtime and routing all traffic to the VM with appropriate modifications to the rules of an ingress traffic controller (e.g., ingress traffic controller 814) (shown in FIG. 10 as 0% traffic to cgroup and namespace-based container runtimes and 100% traffic to VM-based container runtimes).

[0108] The present invention may be embodied as a system, method, or computer program product, or any combination thereof. The computer program product may include one or more computer readable storage media having computer readable program instructions thereon that cause a processor to perform aspects of the present invention.

[0109] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, by way of example and without limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as ridge structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be interpreted as signals that are transitory in nature, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0110] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to an individual computing / processing device or to an external computer or storage device over a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or any combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or any combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage on a computer readable storage medium within the individual computing / processing device.

[0111] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, or the like, and traditional procedural programming languages ​​such as the “C” programming language or similar. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In one or more embodiments, to perform aspects of the invention, electronic circuitry including, by way of example, a programmable logic circuitry, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

[0112] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to one or more embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0113] Such computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the instructions are executed by the processor of the computer or other programmable data processing apparatus to produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. The computer readable program instructions, which can direct a computer, programmable data processing apparatus, or other device, or any combination thereof, to function in a particular manner, may also be stored on a computer readable storage medium such that the computer readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions for implementing each aspect of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0114] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a sequence of operational steps to be performed on the computer, other programmable apparatus, or other device to result in a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0115] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions that implement the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. By way of example, depending on the functionality involved, two blocks shown in succession may in fact be executed virtually simultaneously, or the blocks may be executed in reverse order. Furthermore, it should be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of multiple blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0116] Those skilled in the art will appreciate that numerous variations are possible within the scope of the present invention. By way of example, the specific hardware and software implementation details described herein (i.e., using a Linux® kernel) are for illustrative purposes only and are not intended to limit the scope of the described subject matter. Thus, while the present invention has been shown and described in detail with reference to preferred embodiments thereof, those skilled in the art will appreciate that changes in form and detail may be made thereto without departing from the scope of the present invention.

Claims

1. 1. A method of modifying isolation of containerized workloads in a system having one or more compute nodes, each having a host kernel, comprising: containerizing the workload using a default container runtime, the default container runtime using resource restriction capabilities, including cgroups and namespaces, of a compute node's host kernel to create one or more cgroup-based containers on the compute node; migrating at least a portion of the containerized workloads executing in the one or more cgroup-based containers created by the default container runtime to one or more virtual machines (VMs) launched by a standby container runtime in response to detecting a triggering factor, the triggering factor being selected from the group consisting of a perceived threat, a change in compliance requirements, and combinations thereof; Including, The step of migrating at least a portion of the containerized workload includes: starting the standby container runtime; live migrating the cgroup and the namespace of at least one each container of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs; A method comprising:

2. 10. The method of claim 1, wherein migrating at least a portion of the containerized workload is performed without interruption of service.

3. The method of claim 1 or 2, wherein the default container runtime is a cgroup and namespace based container runtime.

4. The method of claim 1 , wherein the standby container runtime is a VM-based container runtime.

5. wherein live migrating the cgroup and the namespace of at least one each container of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs includes invoking a checkpoint / restore in user space (CRIU), the method comprising: rerouting ingress traffic for the at least some of the migrated containerized workloads to the one or more VM runtimes. The method of claim 1 , further comprising:

6. switching container runtimes of a subset of the containerized workloads running in the one or more cgroup-based containers generated by the default container runtime by starting the workloads in one or more virtual machines (VMs) generated by a standby container runtime, where the switching does not include invoking a CRIU; rerouting ingress traffic for the subset of the switched containerized workloads to the workloads initiated in the one or more VM runtimes; The method of claim 5 further comprising:

7. detecting the trigger event; selecting either a full migration mode or a partial migration mode based on one or more characteristics associated with the triggering factor; The method of claim 1 , further comprising:

8. The step of migrating at least a portion of the containerized workload includes: starting the standby container runtime; live migrating the cgroup and the namespace of each container of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs in response to selecting the full migration mode; The method of claim 7, comprising:

9. The step of migrating at least a portion of the containerized workload includes: starting the standby container runtime; live migrating the cgroup and the namespace of each container of a subset of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs in response to selecting the partial migration mode; The method of claim 7, comprising:

10. 10. The method of claim 9, wherein live migrating the cgroups and the namespaces of each container of a subset of the cgroup-based one or more containers running on the compute node to one or more VM runtimes on the one or more VMs in response to selecting the partial migration mode includes horizontally scaling containers across regular cgroup runtimes and VM runtimes by leveraging load balancing.

11. receiving, at a service mesh controller, runtime metadata transmitted from one or more service mesh sidecars associated with the one or more VM runtimes on the one or more VMs; updating routing rules at the service mesh controller based on the runtime metadata and sending the updated routing rules to an ingress traffic controller; routing ingress traffic at the ingress traffic controller based on the updated routing rules; The method of any of claims 1 to 10, further comprising:

12. A method for modifying isolation of containerized workloads in a system having one or more compute nodes, each having a host kernel, comprising: containerizing the workload using a default container runtime, the default container runtime using resource restriction capabilities, including cgroups and namespaces, of a compute node's host kernel to create one or more cgroup-based containers on the compute node; migrating at least a portion of the containerized workloads executing in the one or more cgroup-based containers created by the default container runtime to one or more virtual machines (VMs) launched by a standby container runtime in response to detecting a triggering factor, the triggering factor being selected from the group consisting of a perceived threat, a change in compliance requirements, and combinations thereof; Including, The step of migrating at least a portion of the containerized workload further comprises: selecting either a full migration mode or a partial migration mode based on one or more characteristics associated with the detected trigger factor; starting the standby container runtime; in response to selecting the partial migration mode, live migrating the cgroup and the namespace for each container of a subset of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs; A method comprising:

13. The method of claim 12, wherein in response to selecting the partial migration mode, live migrating the cgroup and the namespace of each container of a subset of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs includes horizontally scaling containers across normal cgroup runtimes and VM runtimes by leveraging load balancing.

14. A method for providing a service mesh controller, comprising: receiving, at a service mesh controller, runtime metadata transmitted from one or more service mesh sidecars associated with the one or more VM runtimes on the one or more VMs; updating routing rules at the service mesh controller based on the runtime metadata and sending the updated routing rules to an ingress traffic controller; routing ingress traffic at the ingress traffic controller based on the updated routing rules; 14. The method of claim 12 or 13, further comprising:

15. 1. A computer system for modifying isolation of containerized workloads, the computer system comprising: one or more processors, one or more computer readable storage devices, and program instructions stored in at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors. the program instructions comprising: containerizing a workload using a default container runtime, the default container runtime using resource restriction capabilities, including cgroups and namespaces, of a compute node's host kernel to generate one or more cgroup-based containers on the compute node; migrating at least a portion of the containerized workloads executing in the one or more cgroup-based containers created by the default container runtime to one or more virtual machines (VMs) launched by a standby container runtime in response to detecting a triggering factor, the triggering factor being selected from the group consisting of a perceived threat, a change in compliance requirements, and combinations thereof; is feasible for the program instructions executable for migrating at least a portion of the containerized workload include program instructions executable to start the standby container runtime to live migrate the cgroup and the namespace of at least one each of the one or more cgroup-based containers running on the compute node to one or more VM runtimes on the one or more VMs.

16. 16. The computer system of claim 15, further comprising the program instructions executable to reroute ingress traffic for the at least some of the migrated containerized workloads to the one or more VM runtimes.

17. receiving, at a service mesh controller, runtime metadata transmitted from one or more service mesh sidecars associated with the one or more VM runtimes on the one or more VMs; updating routing rules at the service mesh controller based on the runtime metadata and sending the updated routing rules to an ingress traffic controller; Routing ingress traffic at the ingress traffic controller based on the updated routing rules; 17. The computer system of claim 16, further comprising the program instructions executable for:

18. 15. A computer program for modifying isolation of containerized workloads, the computer program causing one or more processors to perform a method according to any of claims 1 to 14.

19. 20. A computer readable medium having stored thereon the computer program of claim 18.

Citation Information

Patent Citations

  • Container-based isolation processing method and relevant equipment

    CN107608757A

  • Methods, systems and programs for determining identities of software in software containers

    JP2017107555A

  • Container deployment based on dependencies

    JP2018530033A

  • Security fix of a container in a virtual machine environment

    US20180053001A1

  • Virtual container deployment

    US20180225104A1