Computer-implemented method, system, and computer program (a new container storage system for remote pods in Kubernetes)

Abstract

To provide a computer-implemented method, system, and computer program product for a novel container storage system in remote pods in Kubernetes (R).SOLUTION: In a container storage system in remote pods, a worker node virtual machine determines that a volume is available for attachment to the worker node virtual machine, and an intermediary software of the worker node virtual machine causes a pod container storage interface to attach the volume to a pod virtual machine. In response to attaching the volume to the pod virtual machine, the intermediary software of the worker node virtual machine causes the pod container storage interface to mount the volume to the pod virtual machine such that the volume is available for use by the pod virtual machine.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates generally to computer systems, and more particularly to a computer-implemented method, computer system, and computer program product configured and arranged for a new container storage system in Kubernetes remote pods. [Background technology]

[0002] Kubernetes, commonly referred to as K8s, is an open-source container orchestration system for automating the deployment, scaling, and management of computer applications. Its purpose is to provide a platform for automating the deployment, scaling, and operation of application containers across a cluster of hosts. Kubernetes works with various container tools, often using Docker to run containers in clusters with images built using Docker. Docker is a family of Platform as a Service (PaaS) products that use operating system (OS)-level virtualization to deliver software in packages called containers. Containers are isolated from each other and bundle their own software, libraries, and configuration files. Containers can communicate with each other through well-defined channels. All containers can share the services of a single operating system kernel. The basic scheduling unit in Kubernetes is the "pod." A pod is a grouping of containerized components. A pod contains one or more containers guaranteed to coexist on the same node. Many cloud services offer Kubernetes-based platforms and infrastructure as services, and Kubernetes can be deployed on top of them as a platform offering. The scheduler is a pluggable component that selects which nodes unscheduled pods (the basic entities managed by the scheduler) run on based on resource availability. The scheduler tracks each node's resource usage and ensures that workloads that exceed the available resources are not scheduled. While existing technology for pods with containers configured to launch and run software applications on nodes in the cloud is suitable for its intended purpose, what is needed is a system having certain features of embodiments of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0003] A computer-implemented method, system, and computer program product are provided for a new container storage system in remote pods in Kubernetes. [Means for solving the problem]

[0004]

[0006] Embodiments of the present invention are directed to a computer-implemented method for a new container storage system in a remote pod in Kubernetes. A non-limiting exemplary computer-implemented method includes determining, by a worker node VM, that a volume is available for attachment to the worker node VM. The computer-implemented method includes using intermediary software in the worker node VM to cause a pod container storage interface to attach the volume to the pod VM. In response to attaching the volume to the pod VM, the computer-implemented method also includes using intermediary software in the worker node VM to cause a pod container storage interface to mount the volume to the pod VM so that the volume is available for use by the pod VM.

[0005] This allows attaching and mounting a volume to a remote pod running on a different virtual machine than the worker node, overcoming the issue of not being able to mount a volume attached to a worker node to a remote pod when the remote pod runs on a different virtual machine than the worker node, and providing an improvement over known methods.

[0006] In addition to or as an alternative to one or more features described above or below, in a further embodiment of the present invention, the volume is a persistent volume, which thus advantageously provides a technique for mounting a persistent volume to a remote pod running on a virtual machine separate from the worker node.

[0007] In addition to or as an alternative to one or more features described above or below, in a further embodiment of the present invention, the worker node virtual machine is configured to receive a request to attach a volume, and instead of attaching the volume to the worker node virtual machine, the request triggers intermediate software to attach the volume to the pod virtual machine. This thus advantageously provides a technique for attaching a volume to a remote pod running on a virtual machine other than the worker node.

[0008] In addition to or as an alternative to one or more features described above or below, in a further embodiment of the present invention, the worker node virtual machine is configured to receive a request to mount a volume, and instead of mounting the volume on the worker node virtual machine, the request triggers intermediate software to mount the volume on the pod virtual machine. This thus advantageously provides a technique for mounting a volume on a remote pod running on a virtual machine other than the worker node.

[0009] Other embodiments of the present invention implement features of the above-described methods in computer systems and computer program products.

[0010] Additional technical features and advantages are realized by the techniques of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, please refer to the detailed description and drawings.

[0011] The particular subject matter of the exclusive rights claimed herein is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a block diagram of an exemplary computer system for use in conjunction with one or more embodiments of the present invention. [Figure 2] FIG. 1 is a block diagram of a system for providing and using remote pods in Kubernetes in accordance with one or more embodiments of the present invention. [Figure 3] FIG. 1 is a block diagram of a node and pod hierarchy in a system for providing and using remote pods as virtual machines in Kubernetes in accordance with one or more embodiments of the present invention. [Figure 4] FIG. 1 is a block diagram of an exemplary architecture for a system for providing and using remote pods as virtual machines in Kubernetes, in accordance with one or more embodiments of the present invention. [Figure 5] 1 is a flowchart of a computer-implemented process for providing and using remote pods as virtual machines for Kubernetes, according to one or more embodiments. [Figure 6] FIG. 1 illustrates an exemplary network architecture for a system for providing and using remote pods as virtual machines in Kubernetes, in accordance with one or more embodiments of the present invention. [Figure 7] 1 is a flowchart of a computer-implemented process for providing and using remote pods as virtual machines in Kubernetes, in accordance with one or more embodiments of the present invention. [Figure 8] FIG. 1 is a block diagram of an exemplary container storage interface architecture in a system for a new container storage system in remote pods in Kubernetes, in accordance with one or more embodiments of the present invention. [Figure 9]FIG. 1 is a block diagram illustrating further architectural details of a system for installing a new container storage system in a remote pod in Kubernetes using one or more new components in a container storage interface driver, in accordance with one or more embodiments of the present invention. [Figure 10] FIG. 1 is a block diagram of a process for installing a new container storage system in a remote pod in Kubernetes using a new container storage interface in the system, in accordance with one or more embodiments of the present invention. [Figure 11] 1 illustrates a block diagram of various functions that can be invoked and / or utilized to install a new container storage system on a remote pod in Kubernetes using one or more new components in a container storage interface driver, in accordance with one or more embodiments of the present invention. [Figure 12] FIG. 1 is a block diagram of an extension container storage interface object for installing a new container storage system in a remote pod in Kubernetes, in accordance with one or more embodiments of the present invention. [Figure 13] 1 is a flowchart illustrating a computer-implemented method for attaching and mounting a persistent volume to a remote pod VM on behalf of a worker node VM, in accordance with one or more embodiments of the present invention. [Figure 14] FIG. 1 illustrates a cloud computing environment in accordance with one or more embodiments of the present invention. [Figure 15] FIG. 2 illustrates an abstraction model layer in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] One or more embodiments of the present invention provide a computer-implemented method, computer system, and computer program product arranged and configured to provide a new container storage system for remote pods in Kubernetes. One or more embodiments of the present invention introduce a new worker and pod hierarchy in which pod VMs logically belong to worker node VMs, but pod VMs physically run in virtual machines remote from worker VMs. Thus, pod VMs have both better isolation and better performance by being in their own virtual machines. Furthermore, one or more embodiments provide the new container storage system by adding a Container Storage Interface (CSI) Pod plugin that can be utilized with a CSI Controller plugin and a CSI Node plugin. According to one or more embodiments, persistent volumes, such as file storage or block storage or a combination thereof, can be attached to remote pod VMs and mounted to containers in the remote pods using the new CSI Pod plugin.

[0014] In a typical Kubernetes cluster, workloads (such as software applications) run within pods as containers, which are processes on worker nodes. Kata Containers, an open-source, Open Container Initiative (OCI)-compliant standard, deploys virtual machines as pods, allowing workloads to run within guest virtual machines within or on worker nodes for cloud-native applications. However, Kata pods running as guest virtual machines on worker node virtual machines can cause performance issues. In particular, second-level or nested virtual machines typically experience a 5-10% decrease in central processing unit (CPU) and memory utilization / efficiency, and input / output (I / O) metrics such as input / output operations per second (IOPS) are even worse. In the worst case scenario, I / O downgrades by approximately 30%.

[0015] One or more embodiments provide a novel system that allows pod virtual machines to run (physically) as peer virtual machines while logically belonging to a worker node virtual machine. In this way, the system aligns with Kubernetes functionality and avoids the performance loss / issues of nested virtual machines in Kata. Technical benefits and advantages, according to one or more embodiments, provide better isolation between worker node virtual machines and remote pod virtual machines because worker nodes do not have direct access to managed remote pod virtual machines. Technical benefits and advantages include better performance of remote pod virtual machines in terms of CPU, memory, and disk usage / performance. Furthermore, one or more embodiments demonstrate better scalability because remote pod virtual machines are not limited by the resources of worker node virtual machines and can be deployed on any host machine.

[0016] Furthermore, because CSI typically attaches persistent volumes to worker nodes first and then mounts the volumes in pods contained within the worker nodes, CSI drivers do not work with remote pod systems. However, remote pod VMs, according to one or more embodiments, run on virtual machines separate from worker node VMs and are thereby not contained within the worker node VMs. Thus, a new container storage system is provided that uses a new CSI pod plugin component in a remote pod VM to attach and mount persistent volumes when the remote pod VM is not contained within the worker node VM. One or more embodiments disclose techniques to support attaching persistent volumes to remote pod VMs, as well as an enhanced VolumeAttachment object for describing the volume attachment.

[0017] FIG. 1 generally illustrates a computer system 100 in accordance with one or more embodiments of the present invention. Computer system 100 may be an electronic, computer framework that includes and / or uses any number and combination of computing devices and networks utilizing various communication technologies, as described herein. Computer system 100 may be easily scalable, extensible, and modular, capable of being adapted to different services, and some functions may be reconfigured independently of other functions. Computer system 100 may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, computer system 100 may be a cloud computing node. Computer system 100 may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. Computer system 100 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0018] As shown in FIG. 1, computer system 100 includes one or more central processing units (CPUs) 101a, 101b, 101c, etc. (collectively or commonly referred to as processors 101). Processor 101 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 101, also referred to as a processing circuit, is coupled to system memory 103 and various other components via system bus 102. System memory 103 may include read-only memory (ROM) 104 and random access memory (RAM) 105. ROM 104 is coupled to system bus 102 and may include a basic input / output system (BIOS) or its successor, such as a Unified Extensible Firmware Interface (UEFI), which controls certain basic functions of computer system 100. RAM is readable and writable memory coupled to system bus 102 for use by processor 101. System memory 103 provides temporary memory space for the execution of instructions during operation. The system memory 103 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0019] Computer system 100 is comprised of input / output (I / O) adapter 106 and communications adapter 107 coupled to system bus 102. I / O adapter 106 may be a small computer system interface (SCSI) adapter that communicates with hard disk 108 or any other similar component or combination thereof. I / O adapter 106 and hard disk 108 are collectively referred to herein as mass storage 110.

[0020] Software 111 for execution on computer system 100 may be stored on mass storage 110. Mass storage 110 is an example of a tangible storage medium readable by processor 101, on which software 111 is stored as instructions for execution by processor 101 to operate computer system 100, as described below with respect to various figures. Examples of computer program products and the execution of such instructions are described in more detail herein. Communications adapter 107 interconnects system bus 102 with network 112, which may be an external network, enabling computer system 100 to communicate with other such systems. In one embodiment, a portion of system memory 103 and mass storage 110 collectively store an operating system, which may be any suitable operating system for coordinating the functions of the various components shown in FIG. 1 .

[0021] Additional input / output devices are shown connected to system bus 102 via display adapter 115 and interface adapter 116. In one embodiment, adapters 106, 107, 115, and 116 may be connected to one or more I / O buses that are connected to system bus 102 via an intermediate bus bridge (not shown). A display 119 (e.g., a screen or display monitor) is connected to system bus 102 by display adapter 115, which may include a graphics controller and a video controller to improve performance of graphics-intensive applications. A keyboard 121, a mouse 122, speakers 123, and the like may be interconnected to system bus 102 via interface adapter 116, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI) and Peripheral Component Interconnect Express (PCIe). Thus, as configured in FIG. 1, computer system 100 includes processing capabilities in the form of processor 101, storage capabilities including system memory 103 and mass storage 110, input means such as keyboard 121 and mouse 122, and output capabilities including speakers 123 and display 119.

[0022] In some embodiments, communications adapter 107 may transmit data using any suitable interface or protocol, such as an Internet Small Computer System Interface, among others. Network 112 may be a cellular network, a wireless network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. External computing devices may connect to computer system 100 via network 112. In some examples, the external computing device may be an external web server or a cloud computing node.

[0023] It should be understood that the block diagram of Figure 1 is not intended to indicate that computer system 100 includes all of the components shown in Figure 1. Rather, computer system 100 may include any suitable fewer or additional components not shown in Figure 1 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, the embodiments described herein with respect to computer system 100 may be implemented with any suitable logic, and logic referred to herein may, in various embodiments, include any suitable hardware (e.g., a processor, embedded controller, or application specific integrated circuit), software (e.g., an application), firmware, or any suitable combination of hardware, software, and firmware.

[0024] FIG. 2 is a block diagram of a system 200 for providing and using remote pods in Kubernetes, in accordance with one or more embodiments of the present invention. System 200 includes computer system 202 coupled to computer systems 251, 252, 253, and 254. Computer system 202 may be referred to as a control node, a master node, a Kubernetes deployment controller, or the like. Computer system 202 may be part of a control plane. Computer systems 251, 252, 253, and 254 may each be a server in a cluster. For purposes of illustration, computer systems 251, 252, 253, and 254 may each host one or more worker nodes in a cluster and may be generally referred to as computer systems 251-254. Computer systems 251-254 may be hosts or host machines for nodes and for running virtual machines, as discussed further herein. Computer system 202 and computer systems 251-254 may include any elements and functionality of computer system 100 discussed in FIG. 1 to perform according to one or more embodiments.

[0025] Kubernetes has a control plane / panel at the top, also known as the master node. After the control plane (e.g., computer system 202) receives workloads, which are applications, the API server 204 stores the applications as API objects in "etcd." In Kubernetes, the controller manager is responsible for orchestration through the control cycle. This control cycle is used for orchestration and helps create the pods required by these applications. When a pod appears, the scheduler monitors changes for new pods. When the scheduler finds a new pod, it helps execute the scheduling algorithm and writes the execution result (e.g., the name of the node in the NodeName field of the pod object, a so-called bind operation). The scheduler then writes the binding result back to etcd, which is the working process of the scheduler. As a result, the pod is bound to a node, which is called scheduling. For example, when a pod is bound to a worker node.

[0026] The kubelet (e.g., kubelet 312 in Figure 3) monitors all changes to pod objects, and when it finds that a pod is bound to a node, the kubelet executes all subsequent tasks. After obtaining this information, the kubelet invokes the containerd process running on each machine to run all containers in this pod. At this time, containerd (e.g., CRI 314 in Figure 3) can invoke other runtimes (e.g., runc, shim, etc.). Therefore, the runtimes can help configure these namespaces and cgroups and help build the containers required by the application.

[0027] FIG. 3 is a block diagram of a node and pod hierarchy in system 200 for providing and using remote pods in Kubernetes, in accordance with one or more embodiments of the present invention. FIG. 3 illustrates worker node virtual machine 310, pod virtual machine 301, pod virtual machine 302, and pod virtual machine 303. Worker node virtual machine 310 can run on the same or different host machines as pod virtual machines 301, 302, and 303, or any combination thereof, where pod virtual machines 301, 302, and 303 can be on the same or different host machines, or any combination thereof. For example, worker node virtual machine 310 and pod virtual machine 301 can run on the same host machine, such as computer system 251. Also, worker node virtual machine 310 and pod virtual machine 301 can run on two different host machines, such as computer systems 251 and 252, respectively. Similarly, worker node virtual machine 310 and pod virtual machine 302 can run on the same host machine (e.g., both on computer system 251) or on different host machines (e.g., one each on computer systems 251 and 253), or a combination thereof. Similarly, worker node virtual machine 310 and pod virtual machine 303 can run on the same host machine (e.g., both on computer system 251) or on different host machines (e.g., one each on computer systems 251 and 254), or a combination thereof. Although each pod virtual machine 301, 302, and 303 may be bound to worker node virtual machine 310, it should be understood that each of pod virtual machines 301, 302, and 303 may run on its own host machine, which may be different from the host machine running worker node virtual machine 310. While three pods are discussed for purposes of illustration, it should be understood that embodiments are not intended to be limited to three pods. Furthermore, each host machine, such as computer systems 251-254, has its own host operating system running its own hypervisor.If pod virtual machines 301, 302, 303 are running on different host machines or different computer systems or combinations thereof than worker node virtual machine 310, the hypervisors of each different host machine or computer system or combination thereof must be in the same logical network.

[0028] The control plane has one or more application programming interface (API) servers 204 configured to communicate with the kubelet 312. The API server 204 (also called an api-server or apiserver) is a lightweight web application that enables Kubernetes to create and expose data APIs from data without the need for custom development. The kubelet is the main "node agent" that runs on each worker node. The kubelet can register nodes with the apiserver using one or more of a hostname, a flag that overrides the hostname, and / or cloud provider-specific logic. The kubelet operates using pod descriptors such as a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. Kubernetes resources are created declaratively, for example, using YAML files. Kubernetes resources such as pods, services, and deployments can be created using YAML files, as will be understood by those skilled in the art. The kubelet receives a set of PodSpecs provided through various mechanisms (primarily through an API server) and ensures that the containers described in those PodSpecs are running and healthy.

[0029] 3, worker node VM 310 has kubelet 312 configured to create one or more shims 320_1, 320_2, 320_3 using container runtime interface (CRI) 314. Shims 320_1, 320_2, 320_3 can be generally referred to as shims 320. Each shim 320_1, 320_2, 320_3 has a one-to-one relationship with its own pod VM 301, 302, 303, respectively. Each individual shim 320_1, 320_2, 320_3 is configured to create its own pod VM 301, 302, 303, such that pod VMs 301, 302, 303 are not nested in worker node VM 310 and / or are guest VMs of worker node VM 310. Container network interface (CNI) 316 is configured to allow the IP network system in pod virtual machines 301, 302, 303 to be managed by the IP network system in worker node virtual machine 310. CRI 314, CNI 316, and shims 320_1, 320_2, 320_3 are each portions of software having computer-executable instructions that can be executed on a processor, such as processor 101.

[0030] CRI314 is sometimes referred to as CRIcontainerd, containerd, or containerdCRI. CRI314 is a high-level runtime and daemon that can be thought of as an API faceplate for other runtimes in the stack. While lower-level runtimes such as runc handle the actual process of running containers, higher-level runtimes such as containerd handle the container lifecycle, image management, and abstraction to the lower-level runtimes. While lower-level runtimes provide the mechanisms for building containers, containerd has the mechanisms for building a container platform and provides APIs used by remote applications for monitoring and delegation. CRI includes APIs, specifications / requirements, and libraries for container runtimes to integrate with the kubelet on a node. CNI316 is a runtime and / or specification for managing network resources on a cluster. CNI consists of a number of supported plugins, as well as a specification and library for writing plugins to configure network interfaces in containers. CNI focuses on container network connectivity and removing allocated resources when a container is deleted. This focus makes CNI a widely supported and easy-to-implement specification. When a Kubernetes component cannot communicate with other components, shim 320 is software configured to translate between that component and the relevant Kubernetes interface. For example, shim 320 takes a CRI command and translates it into something that agent 420 (shown in Figure 4) understands, and vice versa. Additionally, a container runtime shim is a piece of software that sits between the container manager (containerd, cri-o, podman) and the container runtime (runc, crun) and solves the integration issues between these counterparts.

[0031] To provide further detail, FIG. 4 is a block diagram of an exemplary architecture for system 200 for provisioning and using remote pods as virtual machines in Kubernetes, in accordance with one or more embodiments of the present invention. For simplicity, control plane computer system 202 is omitted from FIG. 4 , but it is understood to be present. FIG. 4 illustrates an example of using shim 320_1 in worker node virtual machine 310 to provision, create, and / or set up pod virtual machine 301 remote from worker node virtual machine 310. Pod virtual machine 301 may be hosted on another host machine remote from the host machine running worker node virtual machine 310. Pod virtual machine 301 may be hosted on the same host machine as worker node virtual machine 310, but still be a separate or remote virtual machine. Although an embodiment may discuss using shim 320_1 in worker node virtual machine 310 to set up pod virtual machine 301, it should be understood that the description applies analogously to shim 320_2 for setting up pod virtual machine 302 and shim 320_3 for setting up pod virtual machine 303.

[0032] 5 is a flowchart of a computer-implemented process 500 for providing and using a remote pod as a virtual machine for Kubernetes in accordance with one or more embodiments of the present invention. The computer-implemented process 500 of FIG. 5 can be implemented using system 200. Accordingly, the computer-implemented process 500 will now be described with reference to system 200.

[0033] In block 502, the worker node virtual machine 310 is configured to receive a pod description or pod descriptor for creating a new pod in a cluster on a host machine. For example, the kubelet 312 is configured to receive the pod description from the computer system 202. The pod description includes a pod specification (e.g., PodSpecs) and can be in a YAML file or a JSON file. Each Kubernetes deployment uses a so-called pod template. This pod template provides specifications that determine what a pod should look like, which applications run inside the pod's containers, etc., as will be understood by those skilled in the art. With the assistance of a scheduler, the controller manager of the computer system 202 can match or assign pods to the worker node virtual machine 310.

[0034] In block 504, the worker node virtual machine 310 is configured to monitor the pods and invoke a sandbox function (as an environment) in runtime software (having computer-executable instructions) using the kubelet 312. In one or more embodiments, the shim 320_1 can invoke the sandbox function as a computing environment after the shim 320_1 is started, as discussed further below. A sandbox 460 or pod sandbox is an isolated computing environment on a worker node virtual machine. For example, the kubelet 312 is configured to invoke a sandbox function (e.g., run pod sandbox) of the CRI 314 to start the execution of a pod. A sandbox is a security mechanism for isolating running programs, typically striving to mitigate the spread of system failures or software vulnerabilities, or both. Sandboxes are often used to execute programs or code without risking harm to the host machine or operating system. The kubelet 312 and / or the CRI 314 can invoke the CNI 316.

[0035] In block 506, worker node VM 310 is configured to start and / or create a new shim 320_1 in preparation for the new pod, if one has not already been created above. For example, CRI 314 can invoke shim 320_1 to create a remote virtual machine for the new pod. Shim 320_1 is configured to create / start a shell pod 464 in a pod sandbox (environment) so that shim 320_1 and / or the shell pod appear to be the actual pod virtual machine to worker node VM 310. Shell pod 464 may be an incomplete pod used as a placeholder for the pod virtual machine to be created. More specifically, worker node VM 310 interacts with shim 320_1, which manages / executes pod sandbox 460 with shell pod 464, so that shim 320_1 represents the pod. Worker node virtual machine 310 sees and / or recognizes shim 320_1, which manages / runs pod sandbox 460 with shell pod 464, as the actual pod virtual machine since shell pod 464 is launched like any other pod, but is paused before completing setup.

[0036] In block 508, worker node virtual machine 310 is configured to use shim 320_1 to invoke (pod) virtual machine service 432 and have virtual machine service 432 create / instantiate pod virtual machine 301. Shim 320_1 includes computer-executable instructions for communicating with virtual machine service 432 and requesting that pod virtual machine 301 be created on a host machine. Virtual machine service 432 is a service for the cluster and provides access to endpoints. Because shim 320_1 of worker node virtual machine 310 needs a service endpoint, it invokes virtual machine service 432, which selects a host machine (e.g., a computer system) according to policies such as host resource usage. For example, worker node virtual machine 310 is hosted on computer system 251, and pod virtual machine 301 can be hosted separately on the same computer system 251 (i.e., not nested in worker node virtual machine 310), or on a different computer system 252, or a combination thereof. Unlike state-of-the-art systems for creating pods that reside within the worker node virtual machine itself, in accordance with one or more embodiments of the present invention, shim 320_1 has computer-executable instructions that cause virtual machine service 432 to obtain virtual resources (including CPU, memory, I / O, etc.) outside of worker node virtual machine 310 for creating pod virtual machine 301.

[0037] In block 510, worker node VM 310 is configured to use shim 320_1 to perform pod VM initialization, create / invoke / setup agent 420, create / invoke / setup inner CRI 414 (e.g., inner containerd CRI) in pod VM 301, and set up a network and tunnel between worker node VM 310 and pod VM 301. To set up the network, shim 320_1 is configured to communicate with and / or instruct virtual machine service 432 to cause network handler 434 to create a logical network connection between worker node VM 310 and pod VM 301. This logical network connection is an overlay network that sits on top of the underlay network. Using the overlay network, worker node VM 310 is configured to provide an Internet Protocol (IP) address to pod VM 301. The logical network connection creates a tunnel between worker node VM 310 and pod VM 301 for communication as discussed herein. Worker node virtual machine 310 has a worker node identity, and pod virtual machine 301 has a pod identity, both of which are utilized to create a logical network connection and stored in mapping table 650 of FIG. 6. Through a logical connection from the virtual Ethernet (e.g., eth0) of worker node virtual machine 310 to the virtual Ethernet (e.g., eth0) of pod virtual machine 301, worker node virtual machine 310 and pod virtual machine 301 are on the same network. Network handler 434, in conjunction with the hypervisor, can provide the logical network connection. Network handler 434 runs on a host (e.g., computer system 251) and is responsible for creating an underlay network to connect remote pod virtual machines and worker node virtual machines.If the pod VM and worker node VM are running on two separate hosts, there may be two separate host operating systems and two network handlers working together in the network.

[0038] After setting up the logical connection, shim 320_1 creates agent 420, and agent 420 in pod virtual machine 301 is configured to communicate with shim 320_1 in worker node virtual machine 310. In one or more embodiments, shim 320_1 can invoke agent 420 created in pod virtual machine 301. In one or more embodiments, shim 320_1 can create agent 420 in sandbox 460 and transfer agent 420 to pod virtual machine 301. Agent 420 includes computer-executable instructions that operate as discussed herein. After setting up the logical connection, shim 320_1 creates inner CRI 414, and inner CRI 414 is configured to create a container in pod virtual machine 301. Agent 420 is configured to facilitate forwarding requests between shim 320 and inner CRI 414. In one or more embodiments, shim 320_1 can use agent 420 to invoke inner CRI 414 to be created within pod virtual machine 301. In one or more embodiments, shim 320_1 can create CRI 414 in sandbox 460 and transfer CRI 414 to pod virtual machine 301 via agent 420.

[0039] In block 512, shim 320_1 is configured to invoke / cause inner CRI 414 to create exemplary containers 440 and 441. For example, shim 320_1 is configured to forward APIs and / or API objects for containers intended for shell pod 464 to inner CRI 414 via agent 420, among other things, to pod virtual machine 301 for inner CRI 414 to create containers 440 and 441. Shim 320_1 is configured to send a request to agent 420, which then redirects the request to inner CRI 414 in pod virtual machine 301. CRI 414 is instructed to invoke containerd to create containers 440 and 441. Containers 440 and 441 are each opened / run to start / instantiate another software application on pod virtual machine 301 such that the software application executes on the host machine running pod virtual machine 301. Rather than containers 440 and 441 being created in a shell pod 464 of sandbox 460, containers 440 and 441 are created in pod virtual machine 301. Additionally, suspended container 462 may be launched in sandbox 460. Suspended container 462 may be an incomplete container, a shell of a container, or both.

[0040] In block 514, the customized cAdvisor 410 or the kubelet 312 (the customized cAdvisor 410 may be in the kubelet 312), or a combination thereof, is configured to invoke the resource awareness service 430 to report the pod virtual machine 301 for resource monitoring. Reporting the pod virtual machine 301 includes reporting the pod identity of the pod virtual machine 301 to the resource awareness service 430 and the API server 204. The resource awareness service 430 is configured to maintain the resources of the pod virtual machine 301.

[0041] 6 is a block diagram illustrating an example of a network architecture for system 200 for providing and using remote pods as virtual machines in Kubernetes, in accordance with one or more embodiments of the present invention. System 200 in FIG. 6 omits some elements so as not to obscure the figure, but it should be understood that system 200 includes elements and their functionality as discussed herein. Pod virtual machines 301 and 302 are shown for simplicity, but the description applies to pod virtual machine 303 and any additional pod virtual machines.

[0042] The host network handler 434 is responsible for providing access to an underlay network to set up overlay networks for the worker node virtual machine 310 and the pod virtual machines 301 and 302. The underlay network is the physical infrastructure on which the overlay network is built. The underlay network is the underlying network responsible for delivering packets over the network. The overlay network can be a virtual local area network (VLAN) or any applicable network set up by the CNI 316, or a combination thereof. Each of the pod virtual machines 301 and 302 has a unique pod namespace, as initialized by the shim 320_1 during the creation of the pod virtual machines. For example, the pod virtual machine 301 has a pod namespace 601, and the pod virtual machine 302 has a pod namespace 602. Similarly, the shim 320_1 is configured to create a proxy network namespace 611 and a proxy network namespace 612 for the worker node virtual machine 310. Each pod namespace 601, 602 is an object within a pod virtual machine 301, 302, respectively, and each proxy network namespace 611, 612 is an object within a worker node virtual machine 310. Using the pod identity of pod virtual machine 301 stored in mapping table 650, shim 320_1 assigns proxy network namespace 611 to the proxy or proxy network for pod virtual machine 301. Using the pod identity of pod virtual machine 302 stored in mapping table 650, shim 320_1 assigns proxy network namespace 612 to be the proxy or proxy network for pod virtual machine 302. This mapping is stored in mapping table 650.

[0043] Each pod virtual machine 301, 302 has a virtual Ethernet connection to the pod namespace that contains the container and a virtual Ethernet connection outside the pod namespace. For example, pod virtual machine 301 has a virtual Ethernet connection (VETH1) within pod namespace 601 connected to another virtual Ethernet connection (VETH0) outside the pod namespace 601 within pod virtual machine 301, and the virtual Ethernet connections (VETH1 and VETH0) are a pair. Pod virtual machine 302 has a virtual Ethernet connection (VETH1) within pod namespace 602 connected to another virtual Ethernet connection (VETH0) within pod virtual machine 302 outside the pod namespace 602, and the virtual Ethernet connections (VETH1 and VETH0) are a pair on pod virtual machine 302. In the worker node virtual machine 310, a virtual Ethernet connection (VETH1) in the proxy network namespace 611 is connected to another virtual Ethernet connection (VETH0) outside the proxy network namespace 611, and the virtual Ethernet connections (VETH1 and VETH0) are a pair. In the worker node virtual machine 310, a virtual Ethernet connection (VETH3) in the proxy network namespace 612 is connected to another virtual Ethernet connection (VETH2) outside the proxy network namespace 612, and the virtual Ethernet connections (VETH3 and VETH2) are another pair. The proxy network namespaces 611 and 612 are each connected to a bridge (and have their own IP addresses) via their respective virtual Ethernet connections.

[0044] CNI 316 and / or shim 320_1 are configured to create a tunnel (e.g., tunnel 0) such as a virtual private network between proxy network namespace 611 in worker node virtual machine 310 and pod namespace 601 in pod virtual machine 301, through which traffic (e.g., data) is mirrored / copied to communicate between worker node virtual machine 310 and pod namespace 601. Similarly, CNI 316 and / or shim 320_2 are configured to create another tunnel (e.g., tunnel 1) such as a virtual private network between proxy network namespace 612 in worker node virtual machine 310 and pod namespace 602 in pod virtual machine 301, through which traffic (e.g., data) is mirrored / copied to communicate between worker node virtual machine 310 and pod namespace 602. That is, the tunnel and virtual Ethernet connection mirror traffic back and forth. CNI 316 and / or 320_1 is configured to assign a unique IP address to each of the pod namespaces 601, 602 of pod virtual machines 301, 302, all of which are stored in mapping table 650.

[0045] As discussed herein, pod virtual machines 301, 302, and 303 can be deployed to run on any host machine, which can be different from the host machine running worker node virtual machine 310, or one or more of pod virtual machines 301, 302, and 303 can be on the same host machine as worker node virtual machine 310 and other pod virtual machines can be on different host machines, or a combination thereof. From the worker node's perspective, the fact that the actual container footprints are on a different virtual machine than worker node virtual machine 310 is unaware; that is, there is no CNI code on the pod virtual machines. Furthermore, because worker node virtual machine 310 interfaces with shims 320_1, 320_2, and 320_3 that created the proxy network namespace, it can use the proxy network namespace to send data to and receive data from the pod namespace as if pod virtual machines 301, 302, and 303 were on the same host machine.

[0046] 7 is a flowchart illustrating a computer-implemented method 700 for instantiating / launching and using a remote pod virtual machine that is remote from a worker node virtual machine, in accordance with one or more embodiments of the present invention. The computer-implemented method 700 of FIG. 7 may be implemented using system 200. Reference may be made to FIGS. 1-6, as appropriate.

[0047] At block 702 of the computer-implemented method 700, the worker node virtual machine 310 is configured to instantiate / start / invoke intermediate software (e.g., shim 320) on the worker node virtual machine 310. For example, the CRI 316 (e.g., containerd, etc.) may instantiate / start / invoke the shim 320 on the worker node virtual machine 310.

[0048] In block 704, worker node virtual machine 310 is configured to use intermediate software (e.g., shim 320_1) to create a pod virtual machine (e.g., pod virtual machine 301), where the pod virtual machine (e.g., pod virtual machine 301) is remote from worker node virtual machine 310. For example, pod virtual machine 301 is not nested and / or contained within worker node virtual machine 310. Pod virtual machine 301 may be hosted on the same host machine as worker node virtual machine 310; for example, both may be hosted on computer system 251. Pod virtual machine 301 and worker node virtual machine 310 may be hosted on different host machines; for example, one may be hosted on computer system 251 and the other on computer system 252.

[0049] In block 706, the worker node virtual machine 310 is configured to establish an overlay network between intermediate software (e.g., shim 320_1) in the worker node virtual machine 310 and a pod space (e.g., pod namespace 601) in the pod virtual machine (e.g., pod virtual machine 301).

[0050] In block 708, the worker node virtual machine 310 is configured to use the overlay network to cause containers (e.g., containers 440, 441) to be created within the pod virtual machine (e.g., pod virtual machine 301), and the worker node virtual machine 310 is configured to manage communication with the pod virtual machine (e.g., pod virtual machine 301) using the overlay network.

[0051] The intermediate software (e.g., shim 320_1) is configured to generate an isolated computing environment on the worker node virtual machine 310. For example, shim 320_1 is configured to launch / generate a sandbox 460 as an isolated computing environment on the worker node virtual machine 310.

[0052] The intermediary software (e.g., shim 320_1) is configured to create a proxy network space in an isolated computing environment on a worker node virtual machine. For example, shim 320_1 is configured to initiate / create a proxy network namespace 611 in sandbox 460 as an isolated computing environment on worker node virtual machine 310.

[0053] The intermediary software (e.g., shim 320_1) is configured to connect a logical network between a proxy network space (e.g., proxy network namespace 611) on the worker node virtual machine 310 and a pod space (e.g., pod namespace 601) on the pod virtual machine (e.g., pod virtual machine 310). The logical network (via the overlay network) may be a tunnel, such as a virtual private network, that uses a virtual local area network, virtual Ethernet, or the like for communication between the proxy network namespace 611 on the worker node virtual machine 310 and the pod namespace 601 (having running software applications formed using containers 440).

[0054] An internet protocol (IP) address is assigned to a proxy network space (e.g., proxy network namespace 611) of worker node virtual machine 310, and intermediary software is configured to reassign / move the IP address to a pod space (e.g., pod namespace 601). For example, shim 320_1 is configured to pull an image of a container to be created, and shim 320_1 is configured to create / start container 462 in sandbox 460, which is then paused. Container 462 is stopped and is a shell container. When shim 320_1 notifies CNI 316 that container 462 has been created / started on worker node VM 310, CNI 316 assigns a TCP / IP address to the paused container 462 (e.g., using Classless Inter-Domain Routing (CIDR)), and shim 320_1 is configured to move / assign the TCP / IP address from proxy network namespace 611 (which may be in sandbox 460) in worker node VM 310 to pod VM 301, thereby setting up the network. Thus, pod namespace 601 is assigned the TCP / IP address.

[0055] In response to receiving a container intended for an isolated computing environment on a worker node virtual machine, worker node virtual machine 310 is configured to transfer the container to a pod virtual machine and associate it with a pod namespace. For example, shim 320_1 is configured to transfer containers 440, 441 to pod namespace 601. Intermediary software (e.g., shim 320_1) is configured to cause the software application in container 440, 441 to be instantiated on a pod virtual machine (e.g., pod virtual machine 301). Thus, the software application is configured to run on pod virtual machine 301 remote from worker node virtual machine 310.

[0056] 8 is a block diagram of an exemplary container storage interface architecture in system 200 for a new container storage system in a remote pod for Kubernetes, in accordance with one or more embodiments of the present invention. System 200 in FIG. 8 omits some elements to avoid obscuring the figure, but it should be understood that system 200 includes elements and their functionality as discussed herein. Pod virtual machine 301 is shown for simplicity, but the description applies to pod virtual machines 302, 303, and any additional pod virtual machines.

[0057] FIG. 8 illustrates an example of how a persistent volume 820 in storage device 810 may be attached and mounted to a remote pod, such as remote pod virtual machine 301, while the remote pod virtual machine 301 is running on a virtual machine separate from worker node virtual machine 310. Unlike ephemeral storage and cloud object storage, the persistent volume 820 can be file storage, block storage, or the like, according to one or more embodiments. On-disk files in containers are ephemeral, which poses several challenges for applications running in containers. One challenge is that files are lost when a container crashes; the kubelet restarts the container, but does so in a clean state. Note that, in general, any volumes attached to a worker node virtual machine cannot be mounted to a remote pod virtual machine because they are running on separate virtual machines. However, one or more embodiments are configured to attach and mount the persistent volume 820 to a pod virtual machine 301 using a container storage interface (CSI) controller plug-in 802, a CSI node plug-in 804, a shim 320, and a CSI pod plug-in 806.

[0058] The Container Storage Interface (CSI) is a specification that establishes an industry-standard interface that container orchestration systems (COs) can use to expose arbitrary storage systems to containerized workloads. In-tree refers to code that resides in the Kubernetes core repository. Out-of-tree refers to code that resides outside of the Kubernetes core repository. A CSI volume plugin refers to code that resides somewhere outside of the core Kubernetes repository. A CSI volume driver is an out-of-tree CSI-compatible implementation of a volume plugin that can be used with Kubernetes through the Kubernetes CSI Volume Plugin.

[0059] A persistent volume, or PersistentVolume (PV), is a portion of a cluster's storage that is provisioned administratively or dynamically using a storage class. Persistent volumes are resources within a cluster, similar to how nodes are cluster resources. While persistent volumes are volume plugins, they have a lifecycle independent of the individual pods that use them. This API object captures the details of the storage implementation, such as NFS, iSCSI, or a cloud provider-specific storage system. File storage, block storage, and object storage are storage formats that hold, organize, and present data in different ways, each with their own capabilities and limitations. File storage organizes and represents data as a hierarchical structure of files within folders, block storage divides data into arbitrarily organized and equally sized volumes, and object storage manages data and links it to associated metadata.

[0060] For example, CSI controller plugin 802 of the control plane of computer system 202 is configured to create persistent volume 820 on (hardware) storage 810 and prepare to attach persistent volume 820 to worker node virtual machine 310. CSI controller plugin 802 is configured to invoke CSI node plugin 804 in worker node virtual machine 310, and instead of attaching persistent volume 820 to worker node virtual machine 310, CSI node plugin 804 is configured to invoke shim 320_1 to cause and / or instruct CSI pod plugin 806 to attach persistent volume 820 to pod virtual machine 301. In one or more embodiments, CSI controller plug-in 802 and / or CSI node plug-in 804 can attempt / request to attach persistent volume 820 to shell pod 464 in pod sandbox 460 of worker node VM 310, which triggers shim 320_1 to invoke CSI pod plug-in 806. Accordingly, shim 320_1 causes and / or instructs CSI pod plug-in 806 to attach persistent volume 820 to pod VM 301. After attaching the persistent volume to pod VM 301, shim 320_1 is configured to cause and / or instruct CSI pod plug-in 806 to mount persistent volume 820 to pod VM 301, particularly to containers 440, 441 for use as persistent storage.

[0061] A volume attachment captures an intent to attach and / or detach a specified volume to / from a specified node. Volume attachment objects are non-namespaced. An in-tree CSI volume plugin implements the following internal Kubernetes volume interfaces: (1) to mount / unmount a volume to a specific path, and (2) to attach / detach a volume to a specified node. For mounting and unmounting, the SetUp and TearDown methods of the in-tree volume plugin trigger the NodePublishVolume and NodeUnpublishVolume CSI calls over a Unix domain socket (UDS). Thus, Kubernetes generates a unique target_path (unique per pod per volume) that the CSI plugin passes through NodePublishVolume to mount the volume. Then, once the NodeUnpublishVolume call completes successfully (the volume unmount is verified), Kubernetes deletes the directory.

[0062] FIG. 9 is a block diagram illustrating further details of an architecture in system 200 for installing a new container storage system in a remote pod in Kubernetes using one or more new components in a container storage interface driver, in accordance with one or more embodiments of the present invention. FIG. 10 is a block diagram of a process for installing a new container storage system in a remote pod in Kubernetes using the new container storage interface in system 200, in accordance with one or more embodiments of the present invention. While not shown in FIG. 10 for brevity, there are various functions performed by the control plane in computer system 202 to attach and mount volumes to containers, as would be understood by one of ordinary skill in the art. One or more embodiments further provide a process for attaching a persistent volume 820, such as, for example, file storage or block storage, to a remote pod virtual machine 301 and mounting the persistent volume 820 to one or more containers 440, 441.

[0063] In Figure 9, to deploy a containerized (third-party) CSI volume driver, Figure 9 can be used to create a CSI volume driver container that implements the volume plugin behavior and exposes a gRPC interface over a Unix domain socket with the controller, node, and identity services. Figure 9 can also be used to bundle the CSI volume driver container with optional helper containers (e.g., external attacher, external provisioner, node-driver-registrar, cluster-driver-registrar, external resizer, external snapshotter, liveness probe, etc.) that assist the CSI volume driver container in interacting with the Kubernetes system. More specifically, Figure 9 illustrates creating the following Kubernetes objects to facilitate communication with a Kubernetes controller, StatefulSet, or Deployment with a CSI volume driver container: a Kubernetes-provided container, a cluster-driver-registrar, an external provisioner (required for provision removal / deprovisioning operations), an external attacher (required for attach / detach operations), an external resizer (required for resize operations), an external snapshotter (required for volume-level snapshot operations), and a liveness probe. Figure 9 also illustrates an emptyDir volume mounted by all containers that contain a CSI volume driver. The CSI volume driver container creates its Unix-domain socket in this directory to enable communication with Kubernetes helper container(s). Figure 9 may also include a DaemonSet to facilitate communication with all instances of a kubelet with a CSI volume driver container, a node-driver-registrar responsible for registering the Unix-domain socket with the kubelet, and a liveness probe.

[0064] Referring to FIG. 9, using a control plane in computer system 202, a user, which may be an instance (e.g., an application) and / or an object, is configured to create a persistent volume claim (PVC) object on API server 204 and / or request that API server 204 create a persistent volume claim. A persistent volume claim (PVC) is a request for storage by a user. A persistent volume claim is analogous to a pod. A pod consumes node resources, and a persistent volume claim consumes persistent volume resources. Just as a pod can request a specific level of resources (e.g., CPU and memory), a persistent volume claim can request a specific size and access mode (e.g., access mode can mount ReadWriteOnce, ReadOnlyMany, ReadWriteMany, etc.).

[0065] While the API server 204 is creating the PVC object, the external provisioner is configured to watch / monitor the PVC object. Watching means monitoring changes to the described resource and returning a stream of add, update, and remove notifications. When the external provisioner finds the PVC object on the API server 204, the external provisioner is configured to send a create volume request to the CSI controller plugin 802, for example, using Google® Remote Procedure Call (gRPC) over Unix Domain Sockets (UDS). The CSI controller plugin 802 is configured to create the volume and notify the external provisioner that the volume has been created. The external provisioner is configured to instruct the API server 204 to create a persistent volume (PV) object. The controller manager (e.g., CSI controller plug-in 802) is configured to create a volume attachment object and send the created volume attachment object to the API server 204, and the external provisioner is configured to update the PVC object and bind the PVC object to the volume attachment object so that the PVC object becomes usable. The external attacher is configured to watch / monitor the volume attachment object, and if found, the external attacher is configured to send a controller-published volume to the CSI controller plug-in 802 (e.g., using gRPC over UDS). Thus, the CSI controller plug-in 802 is configured to attach the volume and then notify the external attacher that the volume has been attached to the worker node's virtual machine 310. The external attacher is configured to instruct the API server 204 to update the volume attachment object. Attach / detach operations are also handled by an external component (the "attacher").The attacher monitors the Kubernetes API on behalf of the external CSI volume driver for new volume attachment objects and triggers the appropriate calls to the CSI volume driver to attach the volume. The attacher will monitor the volume attachment object and mark it as attached even if the underlying CSI driver does not support the ControllerPublishVolume call, as Kubernetes has no knowledge of that. While the example diagram has discussed actions taking place in the control plane of computer system 202, it should be understood that more, fewer, and in some cases different actions may occur, as known by those skilled in the art.

[0066] 10 , which describes further details regarding the worker node VM 310 and the remote pod VM 301, the kubelet 312 in the worker node VM 310 is triggered by the attached volume performed by the CSI controller plugin 802, and the kubelet 312 is configured to watch / monitor the volume attachment object in the control plane's API server 204 in action 1002. In action 1004, the kubelet 312 is configured to notify the CSI node plugin 804 to perform the node stage volume (e.g., using gRPC over UDS), and the CSI node plugin 804 is configured to send a createvolumepodattachment to the control plane in action 1006. Instead of the CSI node plug-in 804 mounting the volume on the worker node VM 310, the CSI node plug-in 804 is configured to send the pod stage volume to shim 320_1 in action 1008 and pass the request for the pod stage volume to the CSI pod plug-in 806 in the remote pod VM 301 in action 1010. Thus, upon receiving the pod stage volume request, the CSI pod plug-in 806 is configured to attach the persistent volume 820 to the pod VM 301, which is a volume pod attachment. In action 1012, the CSI pod plug-in 806 is configured to send a get-volume-pod-attachment to the control plane API server 204 via shim 320_1 of the worker node VM 310. Thus, in action 1012, the CSI pod plug-in 806 retrieves the VolumePodAttachment object created in action 1006 from the control plane. "Get" is a method for reading the status of a specified volume attachment.

[0067] 10 , kubelet 312 is configured to notify CSI node plugin 804 to execute node-publish-volume (e.g., using gRPC over UDS), and CSI node plugin 804 is configured to send a create-CSI-pod-volume to API server 204 in the control plane on computer system 202 in action 1016, instead of mounting the volume on worker node VM 310. In particular, in action 1016, a CSIPodVolume object is created in the control plane that describes the volume information and is used in action 1022. In action 1018, kubelet 312 on worker node VM 310 is configured to instruct shim 320_1 to execute a pod sandbox (e.g., sandbox 460), and shim 320_1 can use the pod sandbox (e.g., sandbox 460) as a template and / or computer environment and appear as a location (e.g., file path) for mounting the volume on the proxy and / or (temporary) worker node VM 310. Instead of mounting to the pod sandbox 460, the shell pod 464, or the suspended container 462, or a combination thereof, the shim 320_1 on the worker node VM 310 is configured to publish a pod-published volume to the CSI pod plugin 806 on the pod VM 301 in action 1018, thereby causing the CSI pod plugin 806 to mount the persistent volume 820 to the containers 440, 441 in action 1020. The CSI pod plugin 806 can mount the persistent volume 820 as a CSI pod volume. The CSI pod plugin sends a getCSIPodVolume to the kubelet 312 via the shim 320_1 in action 1022. In action 1022, the CSI pod plugin 806 retrieves the CSIPodVolume object created in action 1016.

[0068] 11 is a block diagram of various functions that can be invoked and / or utilized to install a new container storage system on a remote pod in Kubernetes using one or more new components in a container storage interface driver in accordance with one or more embodiments of the present invention. A CSI controller plug-in 802, a CSI node plug-in 804, and a CSI pod plug-in 806 can perform and / or invoke functions to attach and mount a persistent volume 820 to a container 440, 441 in a pod VM 301, 302, 303, rather than to a worker node VM 310. The CSI controller plug-in 802, the CSI node plug-in 804, and the CSI pod plug-in 806 can be considered CSI drivers.

[0069] 12 is a block diagram of an extended container storage interface object for installing a new container storage system in a remote pod in Kubernetes, according to one or more embodiments of the present invention. "attacher" indicates the name of the volume driver that will handle this request. This is the name returned by "GetPluginName()." "nodeName" is the node to which the volume should be attached. "source" is a "VolumeAttachmentSource" that represents the volume to be attached. A "PersistentVolume" can be attached via an external attacher. The extended container storage interface object can be used and / or implemented by a CSI controller plug-in 802, a CSI node plug-in 804, and a CSI pod plug-in 806, according to one or more embodiments.

[0070] 13 is a flowchart illustrating a computer-implemented method 1300 for attaching and mounting a persistent volume 820 to a remote pod VM on behalf of a worker node VM, in accordance with one or more embodiments of the present invention. The computer-implemented method 1300 of FIG. 13 can be implemented using system 200.

[0071] In block 1302, the worker node VM 310 is configured to determine that a volume (e.g., persistent volume 820) is available to attach to the worker node VM 310. For example, the kubelet 312 of the worker node VM 310 is configured to watch / monitor a volume attachment object (e.g., action 1002 in FIG. 10 ) that serves as a request to attach the persistent volume 820 to the worker node VM 310.

[0072] In block 1304, the worker node virtual machine 310 is configured to use intermediary software (e.g., shim 320) of the worker node virtual machine 310 to cause the pod container storage interface (e.g., CSI pod plugin 806) to attach the volume to the pod virtual machine 301. For example, the shim 320 of the worker node virtual machine 310 is configured to send a request to attach a volume, such as a request for a pod stage volume (e.g., action 1010), to the CSI pod plugin 806, which causes the CSI pod plugin 806 to attach the persistent volume 820 to the pod virtual machine 301.

[0073] In block 1306, in response to attaching the volume to the pod VM 301, the worker node VM 310 is configured to use intermediary software (e.g., shim 320) of the worker node VM to cause a pod container storage interface (e.g., CSI pod plugin 806) to mount the volume (e.g., persistent volume 820) on the pod VM 301 so that the volume is usable by the pod VM 301. For example, the shim 320 is configured to request / instruct the CSI pod plugin 806 to mount the persistent volume 820 on the pod VM 301, for example, using podpublishvolume (e.g., action 1020). Further, the CSI pod plugin 806 can mount the persistent volume 820 on the container 440, 441. The shim 320 can use the pod sandbox 460 as a proxy for the mount request from the kubelet 312 and then send the request / instruction to mount the persistent volume 820.

[0074] The volume is a persistent volume 820. The persistent volume 820 constitutes file storage. The persistent volume 820 constitutes block storage. The worker node VM 310 is configured to receive and / or generate a request to mount the volume (e.g., in action 1004), and instead of attaching the volume to the worker node VM 310, the request triggers intermediate software (e.g., shim 320) to attach the volume to the pod VM 301 (e.g., actions 1008, 1010).

[0075] According to one or more embodiments, worker node virtual machine 310 is configured to receive and / or generate a request to mount a volume (e.g., at action 1014), and instead of mounting the volume on worker node virtual machine 310, the request triggers intermediary software (e.g., shim 320) to mount the volume on pod virtual machine 301 (e.g., actions 1018, 1020). The intermediary software (e.g., shim 320) is configured to use an isolated computing environment (e.g., pod sandbox 460) on worker node virtual machine 310 as a proxy for mounting the volume on pod virtual machine 301.

[0076] Although this disclosure includes detailed descriptions of cloud computing, it will be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0077] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0078] The characteristics are as follows:

[0079] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0080] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0081] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, resulting in a sense of location independence. However, consumers may be able to determine location at a higher level of abstraction (e.g., country, state, data center).

[0082] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly release to instantly scale in. To the consumer, the computing power available for provisioning often appears unlimited, and can be purchased at any time and in any quantity.

[0083] Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0084] The service model is as follows:

[0085] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure that can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, except for limited user-specific application configuration settings.

[0086] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire using programming languages ​​and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0087] Infrastructure as a Service (IaaS): The functionality offered to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which they can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0088] The deployment model is as follows:

[0089] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0090] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0091] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0092] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0093] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0094] FIG. 14 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (e.g., PDAs or cell phones 54A, desktop computers 54B, laptop computers 54C, or automobile computer systems 54N, or combinations thereof) can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers are not required to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 14 are merely exemplary, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0095] A set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 14) is shown in FIG. 15. It should be understood in advance that the components, layers, and functions shown in FIG. 15 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0096] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0097] The virtualization layer 70 provides an abstraction layer from which the following virtual entities can be provided, for example: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0098] By way of example, the management layer 80 may provide the following functionality: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future cloud computing resources required in accordance with SLAs.

[0099] The workload layer 90 provides examples of functionality available to a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and software applications 96 implemented in the workloads and functionality (e.g., software applications in worker node virtual machines 310, software applications in pod virtual machines 301, 302, 303, etc.).

[0100] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternate embodiments of the present invention may be devised without departing from the scope of the present invention. In the following description and in the drawings, various connections and relationships (e.g., above, below, adjacent, etc.) between elements are shown. These connections and / or relationships may be direct or indirect unless otherwise specified, and the present invention is not intended to be limited in this respect. Thus, coupling of entities may refer to either direct or indirect coupling, and relationships between entities may be direct or indirect. Furthermore, various tasks and process steps described herein may be combined into a more comprehensive procedure or process having additional steps or functions not specifically described herein.

[0101] One or more of the methods described herein can be implemented with any one or combination of the following technologies known in the art: discreet logic circuits having logic gates for implementing logical functions on data signals, application specific integrated circuits (ASICs) having appropriate combinations of logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0102] For the sake of brevity, conventional techniques related to making and using aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing various technical features described herein are well known. Thus, for the sake of brevity, many conventional implementation details are only briefly mentioned herein or are omitted entirely without providing details of well-known systems and / or processes.

[0103] In some embodiments, various functions or acts may be performed at a given location, or in connection with the operation of one or more devices or systems, or both. In some embodiments, a portion of a given function or act may be performed at a first device or location, and the remainder of the function or act may be performed at one or more additional devices or locations.

[0104] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "comprises" and / or "comprising" specifies the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, element components, or groups or combinations thereof.

[0105] Corresponding structures, materials, acts, and equivalents of all means- or step-plus-function elements in the following claims are intended to include any structure, material, or act for performing the function in combination with the elements of other claims that are specifically claimed. This disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosure. The embodiments were chosen and described to best explain the principles and practical applications of the disclosure, and to enable those skilled in the art to appreciate the disclosure in various embodiments with various modifications as suited to the particular uses contemplated.

[0106] The diagrams shown herein are illustrative. There may be many variations to the diagrams or the steps (or operations) described therein without departing from the spirit of the disclosure. For example, operations may be performed in a different order, or operations may be added, deleted, or modified. Also, the term "coupled" describes having a signal path between two elements, and does not imply a direct connection with no intervening elements / connections between the elements. All of these variations are considered part of the disclosure.

[0107] The following definitions and abbreviations will be used in interpreting the claims and the specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," "containing," or any other variation thereof, are intended to be exclusive inclusions. For example, a composition, mixture, process, method, article, or device consisting of a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed in such composition, mixture, process, method, article, or device.

[0108] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include one or more, i.e., any integer number such as 1, 2, 3, 4, etc. The term "plurality" is understood to include two or more, i.e., any integer number such as 2, 3, 4, 5, etc. The term "connected" can include both indirect and direct "connections."

[0109] The terms "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with measurement of a particular quantity based on equipment available at the time of filing. For example, "about" can include a range of ±8%, 5%, or 2% of a particular value.

[0110] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0111] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0112] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0113] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0114] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer-readable program instructions.

[0115] The computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having the instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0116] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0117] The flowcharts and block diagrams in the figures of this disclosure illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing specific logical functions. In some implementations, the functions shown in the blocks may be executed in a different order than shown in the figures. For example, two blocks shown in succession may actually be executed concurrently or nearly concurrently, or may even be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs specific functions or operations or executes a combination of dedicated hardware and computer instructions.

[0118] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. determining, by a worker node virtual machine, that a volume is available for attachment to said worker node virtual machine; using intermediate software on the worker node virtual machine to cause a pod container storage interface to attach the volume to the pod virtual machine; In response to attaching the volume to the pod virtual machine, using the intermediate software on the worker node virtual machine to cause the pod container storage interface to mount the volume to the pod virtual machine so that the volume is available to the pod virtual machine.

2. The computer-implemented method of claim 1 , wherein the volume is a persistent volume.

3. The computer-implemented method of claim 1 , wherein the volume comprises a file storage.

4. The computer-implemented method of claim 1 , wherein the volume comprises block storage.

5. the worker node virtual machine is configured to receive a request to attach the volume; 2. The computer-implemented method of claim 1, wherein instead of attaching the volume to the worker node virtual machine, the request triggers the intermediary software to attach the volume to the pod virtual machine.

6. the worker node virtual machine is configured to receive a request to mount the volume; 2. The computer-implemented method of claim 1, wherein instead of mounting the volume on the worker node virtual machine, the request triggers the intermediary software to mount the volume on the pod virtual machine.

7. 2. The computer-implemented method of claim 1, wherein the intermediary software is configured to use an isolated computing environment on the worker node virtual machine as a proxy for mounting the volume on the pod virtual machine.

8. a memory having computer readable instructions; one or more processors for executing the computer-readable instructions, the computer-readable instructions causing the one or more processors to: determining, by a worker node virtual machine, that a volume is available for attachment to said worker node virtual machine; using intermediate software on the worker node virtual machine to cause a pod container storage interface to attach the volume to the pod virtual machine; In response to attaching the volume to the pod virtual machine, using the intermediate software of the worker node virtual machine to cause the pod container storage interface to mount the volume to the pod virtual machine so that the volume is available to the pod virtual machine.

9. The system of claim 8 , wherein the volume is a persistent volume.

10. The system of claim 8 , wherein the volume comprises a file storage.

11. The system of claim 8 , wherein the volume comprises block storage.

12. the worker node virtual machine is configured to receive a request to attach the volume; 9. The system of claim 8, wherein instead of attaching the volume to the worker node virtual machine, the request triggers the intermediary software to attach the volume to the pod virtual machine.

13. the worker node virtual machine is configured to receive a request to mount the volume; 10. The system of claim 8, wherein instead of mounting the volume on the worker node virtual machine, the request triggers the intermediary software to mount the volume on the pod virtual machine.

14. 10. The system of claim 8, wherein the intermediary software is configured to use an isolated computing environment on the worker node virtual machine as a proxy for mounting the volume on the pod virtual machine.

15. A computer comprising: determining, by a worker node virtual machine, that a volume is available for attachment to said worker node virtual machine; using intermediate software on the worker node virtual machine to cause a pod container storage interface to attach the volume to the pod virtual machine; In response to attaching the volume to the pod virtual machine, using the intermediate software of the worker node virtual machine to cause the pod container storage interface to mount the volume to the pod virtual machine so that the volume is available to the pod virtual machine.

16. The computer program product of claim 15 , wherein the volume is a persistent volume.

17. The computer program product of claim 15 , wherein the volume comprises a file storage.

18. The computer program product of claim 15 , wherein the volume comprises block storage.

19. the worker node virtual machine is configured to receive a request to attach the volume; 16. The computer program product of claim 15, wherein the request triggers the intermediary software to attach the volume to the pod virtual machine instead of attaching the volume to the worker node virtual machine.

20. the worker node virtual machine is configured to receive a request to mount the volume; 16. The computer program product of claim 15, wherein the request triggers the intermediary software to mount the volume on the pod virtual machine instead of mounting the volume on the worker node virtual machine.

Images