Computer implementation methods, systems, and computer programs (dynamic scaling for workload execution)

Abstract

To solve a problem of unlimited / unplanned cloud expansion resulting in significant threats to IT managers who are concerned about considerations such as a management cost, a charge-back, a capacity, and a resource.SOLUTION: An aspect of the present invention includes the steps of: receiving a workload including one or more tasks by a controller; generating a first pod including a first sidecar container; generating one or more temporary containers for the first pod based on one or more resource assignment metrics relating to the workload and the pod; executing one or more tasks in the one or more temporary container; monitoring the one or more resource assignment metrics relating to the pod; and generating at least one new temporary container in the first pod based on the one or more resource assignment metrics related to the pod and the workload.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention generally relates to data management, and more particularly to dynamic scaling of workload execution in a cloud environment.

Background Art

[0002] Cloud computing refers to the act of using a network of remote servers hosted on a public network (e.g., the Internet) rather than on local servers to provide information computing services (i.e., cloud services). The network architecture (e.g., a virtualized information processing environment comprising hardware and software) that provides these cloud services to service users (i.e., cloud service users) is referred to as the "cloud", which can be a public cloud (e.g., cloud services publicly provided to cloud service users), a private cloud (e.g., a private network or data center that supplies cloud services only to a specific group of cloud service users within an enterprise), or a community cloud (e.g., a set of cloud services publicly provided to a limited set of cloud service users, e.g., an institution having a specific state / region or set of states / regions), a dedicated / hosted private cloud, or other emerging cloud service delivery models. The underlying intent of cloud computing is to provide cloud service users with easy and scalable access to computing resources and information technology (IT) services.

[0003] Cloud services can be broadly divided into four categories: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), and Managed Services. Infrastructure as a Service refers to virtualized computing infrastructure through which cloud services (e.g., virtual server space, network connectivity, bandwidth, IP addresses, load balancers, etc.) are provided. Platform as a Service in the cloud refers to a set of software and product development tools hosted on the cloud that enable developers (i.e., a type of cloud service user) to build applications and services using the cloud. Software as a Service refers to applications hosted by the cloud and available on demand to cloud service users. Managed Services refer to services such as backup management, remote system management, application management, and security services that are enabled by a managed service provider for any given cloud service.

[0004] Generally, cloud services have three distinct characteristics that distinguish them from traditionally hosted services. The first is that they are sold to service users on demand (e.g., by the minute or hour). The second is that they are dynamic (e.g., service users can receive as much service as they want at any given time). The third, and this applies specifically to public clouds rather than private or hybrid clouds, is that the service is fully managed by the cloud service provider (e.g., service users only need suitably equipped client devices and network connectivity). This third characteristic is particularly relevant to public clouds. However, private clouds may be managed by an internal IT department or through an ITO (IT Outsourcing) contract. In these examples, the I&O (Infrastructure & Operations) administrator acts as the cloud provider, and therefore this third characteristic has similar relevance.

[0005] The cloud is rapidly being adopted by businesses and IT users as a way to enhance organizational effectiveness and save costs. Along with this opportunity, however, several new challenges and significant risks arise for businesses to address. For example, business users are rapidly investing in their own cloud capabilities (e.g., IaaS, PaaS, and SaaS) to meet their business needs, and application developers want to move quickly without involving IT in provisioning tools and environments. [Overview of the project] [Problems that the invention aims to solve]

[0006] These developments pose a significant threat to IT administrators who are concerned about considerations such as management costs, chargebacks, capacity, and resources, as a result of unlimited / unplanned cloud expansion. [Means for solving the problem]

[0007] Embodiments of the present invention relate to the dynamic scaling of workload execution in a cloud environment. Non-limiting examples of computer implementation methods include the steps of: a controller receiving a workload having one or more tasks; generating a first pod having a first sidecar container; generating one or more ephemeral containers for the first pod based on the workload and one or more resource allocation metrics relating to the pod; executing one or more tasks within the one or more ephemeral containers; monitoring one or more resource allocation metrics relating to the pod; and generating at least one new ephemeral container within the first pod based on one or more resource allocation metrics relating to the pod and the workload.

[0008] Another embodiment of the present invention implements the features of the above method in a computer system and a computer program product.

[0009] Further technical features and benefits are realized by the technology of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered to be part of the subject matter claimed. For a better understanding, please refer to the detailed description and drawings. [Brief explanation of the drawing]

[0010] Details of the exclusive rights described herein are specifically set forth and expressly claimed in the claims at the end of the specification. The aforementioned and other features and advantages of embodiments of the present invention are evident from the following detailed description, which should be read in conjunction with the accompanying drawings.

[0011] [Figure 1] This document illustrates a cloud computing environment according to one or more embodiments of the present invention.

[0012] [Figure 2]Shows an abstraction model layer according to one or more embodiments of the present invention.

[0013] [Figure 3] Shows a block diagram of a computer system for use in implementing one or more embodiments of the present invention.

[0014] [Figure 4] Shows a compute node using full virtualization and OS-level virtualization.

[0015] [Figure 5] Shows a compute node using full virtualization and OS-level virtualization.

[0016] [Figure 6] Shows a block diagram of a Kubernetes pod according to one or more embodiments of the present invention.

[0017] [Figure 7] Shows a block diagram of a pod manager that performs resource management of pods according to one or more embodiments of the present invention.

[0018] [Figure 8] Shows a block diagram of a pod manager and a horizontal pod autoscaler according to one or more embodiments of the present invention.

[0019] [Figure 9] Shows a block diagram of a controller module communicating with a pod manager and a workload queue according to one or more embodiments of the present invention.

[0020] [Figure 10] Shows a system for dynamically scaling workload execution in a cloud environment according to one or more embodiments of the present invention. [[ID=~51]]

[0021] [Figure 11]A flowchart of a method for dynamically scaling workload execution in a cloud environment according to one or more embodiments of the present invention is shown.

[0022] The figures shown herein are illustrative. Without departing from the spirit of the present invention, many changes can be made to the figures or the operations described therein. For example, actions can be performed in a different order, or actions can be added, deleted, or modified. Also, the term "coupled" and its variations represent that there is a communication path between two elements, and do not imply a direct connection between elements without intervening elements / connections between them. All of these variations are considered to be part of this specification.

Embodiments for Carrying Out the Invention

[0023] One or more embodiments of the present invention provide systems and methods for dynamically scaling workload execution in a cloud environment. The cloud environment may be a platform that manages containerized workloads and services, such as Kubernetes. Kubernetes coordinates computing, networking, and storage infrastructure on behalf of user workloads. Aspects of the present invention provide systems and processes for scaling execution environments (pods) on a Kubernetes platform. A pod is the basic building block of Kubernetes, i.e., the smallest and simplest unit in the Kubernetes object model that is created and deployed. A pod represents an execution process in a distributed computing environment, such as a computer cluster. A pod encapsulates an application container (or possibly multiple containers) and includes storage resources, a unique network IP, and options to control how the container should run. A pod represents a single instance of an application in Kubernetes, which can consist of a unit of deployment, i.e., a single container, or a small number of tightly coupled containers sharing resources. Further embodiments of the present invention allow for the creation of multiple containers in a given pod based on resource allocation metrics monitored for that pod. These multiple containers enable tasks to run in parallel without the need to constantly create a new pod or container, or a combination thereof, for each task. Furthermore, for larger workloads, additional pods containing these multiple containers can be created, managed by the pod manager and controller, to run multiple tasks in parallel. If available resources change, pods or containers, or combinations thereof, can be created or terminated, or both. This makes it possible to scale pods and containers to run tasks in parallel and maximize task execution.

[0024] A new information technology (IT) delivery model is cloud computing, which delivers shared resources, software, and information on demand to computers and other devices over the internet. Cloud computing can significantly reduce IT costs and complexity, while improving workload optimization and service delivery. This approach allows application instances to be hosted and made available from internet-based resources accessible via traditional web browsers over HTTP. An exemplary application might provide a common set of messaging functions such as email, calendars, contact management, and instant messaging. Users then access the service directly over the internet. By using this service, companies can place their email, calendar, or collaboration infrastructure, or a combination thereof, in the cloud, and end users can access their email or perform calendar operations using the appropriate client.

[0025] Cloud compute resources are typically housed in large server farms running one or more network applications, using a virtualization architecture that runs applications within so-called "virtual machines" (VMs) that are usually mapped onto virtual servers or physical servers within data center facilities. Virtual machines typically run on a hypervisor, which is a control program that allocates physical resources to virtual machines. Modern hypervisors often use hardware-assisted virtualization, which provides efficient and complete virtualization by primarily utilizing dedicated virtualization hardware functions from the host CPU.

[0026] Operating system (OS) level virtualization is another approach to virtualization. OS-level virtualization allows for the partitioning of computer resources through the support of the operating system kernel for multiple isolated user-space instances, commonly called containers. Therefore, this virtualization approach is often referred to as container-based virtualization. Containers can be indistinguishable to end users from separate machines. Container-based virtualization is widely used in cloud environments. For example, in many current data centers, workloads run within containers. Containers can provide better agility and orchestration to the changing demands of workloads. Many technologies are used in container-based virtualization. These technologies include, for example, namespaces and cgroups, which will be discussed later.

[0027] In typical container cloud computing environments, the host runs an operating system such as the Linux® kernel. As mentioned above, the term "container" refers to an OS-level virtualization mechanism for running isolated computing workloads (containers) on a control host using a single operating system kernel. This approach effectively partitions resources managed by a single operating system into isolated groups, better balancing the conflicting demands for resource usage between these isolated groups. In contrast to other types of virtualization, it does not require instruction-level emulation or just-in-time compilation. In addition, containers can execute instructions native to the core CPU without any special interpretation mechanism. The operating system gives applications the illusion of running on separate machines while sharing many of the underlying resources, by providing a way to create and enter containers.

[0028] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0029] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0030] The characteristics are as follows:

[0031] On-demand self-service: Cloud users can unilaterally provision computing functions such as server time and network storage automatically as needed, without requiring human interaction with the service provider.

[0032] Broad network access: Multiple functions are available over the network and accessed through standard mechanisms that facilitate use by different types of thin client platforms or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0033] Resource pooling: A provider's computing resources are pooled to serve multiple users using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Users generally have no control or knowledge of the exact location of the resources provided, but there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0034] Rapid Adaptability: Multiple functions can be provisioned quickly, flexibly, and sometimes automatically, scaled out instantly, and released rapidly to scale in instantly. To the user, it often appears as if there are unlimited functions available for provisioning, and they can be purchased at any amount at any time.

[0035] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging appropriate metric capabilities for service types (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Monitoring, controlling, and reporting resource usage provides transparency to both service providers and users.

[0036] Infrastructure as a Service (IaaS): The functionality provided to the user is the provisioning of processing, storage, networking, and other fundamental computing resources, allowing the user to deploy and run any software, which may include operating systems and applications. The user does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage, and deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0037] The deployment model is as follows:

[0038] Private Cloud: A cloud infrastructure is operated exclusively for a specific organization. A private cloud may be managed by that organization or a third party, and may reside on-premises or off-premises.

[0039] Community Cloud: A cloud infrastructure is shared by several organizations to support a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by these organizations or third parties, and may reside on-premises or off-premises.

[0040] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0041] Hybrid Cloud: Cloud infrastructure is a configuration consisting of two or more clouds (private, community, or public) where each cloud remains an independent entity, but they are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing across clouds).

[0042] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0043] Referring here to Figure 1, an exemplary cloud computing environment 50 is shown. As illustrated, the cloud computing environment 50 comprises one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud users, such as personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof. The nodes 10 can communicate with each other. They may be physically or virtually grouped within one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or a combination thereof, as described above (not shown). This enables the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, for which cloud users do not need to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N shown in Figure 1 are for illustrative purposes only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network or network-addressable connection (e.g., using a web browser) or both.

[0044] Referring now to Figure 2, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 1) is shown. It should be understood that the components, layers, and functionalities shown in Figure 2 are for illustrative purposes only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0045] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include a mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based servers 62, 63, blade servers 64, storage devices 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0046] The virtualization layer 70 provides an abstraction layer from which examples of the following virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0047] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 dynamically procures computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 tracks the costs of using resources within the cloud computing environment and charges or bills for the consumption of these resources. In one example, these resources may include application software licenses. Security verifies the identity of cloud users and tasks, and protects data and other resources. The user portal 83 provides access to the cloud computing environment for users and system administrators. Service level management 84 allocates and manages cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and execution 85 pre-arranges and procures cloud computing resources where future requirements are expected to comply with SLAs.

[0048] The workload layer 90 provides examples of capabilities that can be utilized in a cloud computing environment. Examples of workloads and capabilities that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and dynamic scaling 96 for workload execution.

[0049] Referring to Figure 3, one embodiment of a processing system 300 for carrying out the teachings of this specification is shown. The processing system 300 can be implemented as a computing node 10. Furthermore, some or all of the functions of the elements of system 300 may be implemented as a node 10 of a cloud 50 (shown in Figures 1 and 2). The cloud computing node 10 is merely an example of a suitable cloud computing node and is not intended to imply any limitation on the use or scope of functionality of the embodiments of the invention described herein. In this embodiment, system 300 has one or more central processing units (processors) 21a, 21b, 21c, etc. (collectively or generally referred to as processor 21). In one or more embodiments, each processor 21 may include a reduced instruction set computer (RISC) microprocessor. The processor 21 is coupled to system memory 34 and various other components via a system bus 33. Read-only memory (ROM) 22 is coupled to the system bus 33 and may include a basic input / output system (BIOS) that controls certain basic functions of system 300.

[0050] Figure 3 further shows an input / output (I / O) adapter 27 and a network adapter 26 coupled to the system bus 33. The I / O adapter 27 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 23 or a tape storage drive 25 or both, or any other similar component. The I / O adapter 27, the hard disk 23, and the tape storage device 25 are collectively referred to herein as mass storage 24. The operating system 40 running on the processing system 300 may be stored in the mass storage 24. The network adapter 26 interconnects the bus 33 to an external network 36, enabling the data processing system 300 to communicate with other such systems. A screen (e.g., a display monitor) 35 is connected to the system bus 33 by a display adapter 32, which may include a graphics adapter and a video controller to improve the performance of graphics-intensive applications. In one embodiment, adapters 27, 26, and 32 may be connected to one or more I / O buses connected to the system bus 33 via an intermediate bus bridge (not shown). A suitable I / O bus for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically includes a common protocol such as the Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to the system bus 33 via a user interface adapter 28 and a display adapter 32. The keyboard 29, mouse 30, and speaker 31 are all interconnected to the bus 33 via the user interface adapter 28, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit.

[0051] In an exemplary embodiment, the processing system 300 includes a graphics processing unit 41. The graphics processing unit 41 is a dedicated electronic circuit designed to manipulate and modify memory to accelerate the creation of images in a frame buffer intended for output to a display. Generally, the graphics processing unit 41 is highly efficient in computer graphics manipulation and image processing and has a highly parallel structure, making it more effective than a general-purpose CPU for algorithms where large blocks of data are processed in parallel.

[0052] Therefore, when configured as shown in Figure 3, the system 300 includes processing functions in the form of a processor 21, storage functions including system memory 34 and mass storage 24, input means such as a keyboard 29 and a mouse 30, and output functions including a speaker 31 and a display 35. In one embodiment, a portion of the system memory 34 and mass storage 24 collectively store the operating system and coordinate the functions of the various components shown in Figure 3.

[0053] Figures 4 and 5 show compute nodes 400 and 500 using full virtualization and OS-level virtualization, respectively. Several embodiments of the present invention can be used with any of these types of compute nodes, and can also be used in hybrid environments in which these compute nodes are combined, either as a single or multiple compute nodes.

[0054] As shown in Figure 4, each compute node 400 includes hardware 406, which may include a processor (or CPU) 407, memory 408, a network interface card (NIC) 409, and a disk drive 410. The disk drive 410 may include a solid-state drive, a hard disk drive, or any combination of the two. The compute node 400 runs a host operating system 405 on the hardware. The compute node 400 also includes a hypervisor 404 for sharing and managing the hardware 406, allowing multiple different environments 401, isolated from each other, to run on the same physical machine 400. The hypervisor 404 may use hardware-assisted virtualization, which provides efficient and complete virtualization primarily by using dedicated virtualization hardware capabilities from the host CPU 407. Each compute node 400 includes one or more virtual machines 401, each including a guest operating system 403 and one or more application programs (or applications) 402 that run on the guest operating system 403.

[0055] Similarly, as shown in Figure 5, each compute node 500 includes hardware 506, which may include a processor (or CPU) 507, memory 508, a network interface card (NIC) 509, and a disk drive 510. The disk drive 510 may include a solid-state drive, a hard disk drive, or any combination of the two. The compute node 500 runs a host operating system 505 on the hardware. Each compute node 500 includes one or more containers 501, each of which includes one or more applications 502.

[0056] According to some embodiments, a compute node 500 may contain one or more pods 503, each containing one or more containers 501, each containing one or more applications 502. For example, in Kubernetes, containers run within pods.

[0057] Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services. It facilitates both declarative configuration and automation. Kubernetes orchestrates compute, networking, and storage infrastructure on behalf of user workloads. Kubernetes is an example of an orchestration framework. Containerized workloads across multiple compute nodes can be managed by a Container Orchestration Manager (COM). An example of a Container Orchestration Manager (COM) is the Kubernetes Master. Several binary components (e.g., master components, node components, and add-ons) are used to provide a functional Kubernetes cluster. The master component provides the control plane of the Kubernetes cluster (also referred to as the "Kubernetes control plane"). The master component may include, but is not limited to, kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager. The master component makes comprehensive decisions about the Kubernetes cluster. For example, the master component handles scheduling. In addition, the master component is used in detecting and responding to cluster events. For example, the master component is responsible for launching new pods if the "replica" field of the replication controller is not filled. The master component can run on any machine in the cluster. However, setup scripts typically start all master components on the same machine and do not run user containers on that machine.

[0058] Node components run on any compute node in a Kubernetes cluster. Node components are responsible for maintaining the running pods and providing the Kubernetes runtime environment. Node components may include, but are not limited to, kubelet, kube-proxy, and the container runtime. kubelet is an agent that verifies that containers are running within pods. It ensures that containers specified in a set of PodSpecs, provided through various mechanisms, are running and healthy. kube-proxy is a network proxy. It enables Kubernetes service abstractions by maintaining network rules to compute nodes and performing connection forwarding.

[0059] A container runtime is the software responsible for running containers. More specifically, a container runtime is a node component that handles the container lifecycle. It implements fundamental concepts such as creating, starting, stopping, and removing container workloads. Kubernetes supports several runtimes, including but not limited to Docker, containerd, CRI-O, and rktlet.

[0060] More generally, Kubernetes supports any implementation of the Container Runtime Interface (CRI) provided by Kubernetes. CRI enables easy plugging in of various container runtimes. Before CRI was introduced in Kubernetes 1.5, only the default Docker image repository and its default OCI-compliant runtime, runC, were used. The Open Container Initiative (OCI) created runtime specifications detailing the APIs of OCI-compliant container runtimes. runC, runV, and Intel's Clear Containers (also known as "cc-runtime") are examples of OCI-compliant container runtimes. runC has built-in support for CRIU, i.e., user-space checkpoint / restore, for checking and restoring containers, as described below. runV is a hypervisor-based Docker runtime for OCI. runV is also known as "Hyper runV".

[0061] The CRI runtime is at a higher level of abstraction and should not be confused with the OCI-compliant runtime. The CRI runtime is also referred to as a "CRI shim." CRI shims include cri-containerd, CRI-O, dockershim, and franki. Some CRI shims (e.g., cri-containerd, CRI-O, and dockershim) invoke the OCI-compliant runtime, while others (e.g., franki) are monolithic solutions.

[0062] At least some CRI shims support multiple runtimes running on a single compute node. For example, CRI-O supports the concepts of trusted and untrusted sandboxes. In Kubernetes, one or more VM-based pods and one or more cgroup / namespace-based pods can run on a single compute node, based on pod annotations and the default CRI-O configuration. Similar to what is done by runC, containers running within VM-based pods can be isolated and managed via namespaces and cgroups.

[0063] Add-ons are pods and services responsible for implementing cluster functionality. Add-ons include, but are not limited to, cluster DNS (i.e., DNS servers that provide DNS records for Kubernetes services), dashboards (i.e., web-based UIs for Kubernetes clusters that enable users to manage and troubleshoot applications running within the cluster as well as the cluster itself), container resource monitoring (i.e., responsible for recording common time-series metrics about containers in a central database and providing a UI for browsing the data recorded in that database), and cluster-level logging (i.e., responsible for storing container logs in a central log store with a search / browse interface).

[0064] Referring here to an overview of the technology more specifically relevant to the aspects of the present invention, a Kubernetes platform for managing containerized workloads and services requires a startup process to launch pods containing one or more containers. This pod startup process consists of three stages. The first stage is the initialization / preparation stage. The second stage is the creation of a sandbox container by the Container Runtime Interface (CRI). The final (third) stage is the creation of the actual container. The first and second stages can take a considerable amount of time (e.g., 2-3 seconds). A Kubernetes pod must go through this process end-to-end for each task, which can take a considerable amount of time to run a short workload. There are two exemplary cases where this time can cause problems. The first case is when instantaneous throughput in a cloud environment is high, and it is necessary to start one or more pods in a Kubernetes cluster for each user logic. This results in starting a large number of similar pods. These similar pods may be for similar short workloads, such as API calls. Since it takes 2-3 seconds or more to start a pod, it occupies a large portion of the overall workload. The second case is when there is a pipeline, and one stage may contain multiple tasks, and similarly, Kubernetes needs to start a pod for each task. If the number of tasks in the pipeline is large, it may not be possible to start all the pods due to hardware limitations.

[0065] One or more embodiments of the present invention address the aforementioned shortcomings of the prior art by providing systems and processes for dynamically scaling workload execution in a cloud environment. Multiple embodiments include dynamically starting multiple containers to run a workload in a pod of a Kubernetes cluster. Resource metrics and statistics can be collected for the resources allocated for the execution of workloads in these pods and containers. Based on the workload queue, pods can be scaled up, scaled down, or both, relative to other nodes in the Kubernetes cluster. Containers can then be associated with workloads and monitored while the workload is running.

[0066] Figure 6 shows a block diagram of a Kubernetes pod according to one or more embodiments of the present invention. A Kubernetes pod represents an execution process in a distributed computing environment, such as a computer cluster. A pod encapsulates an application container (or possibly multiple containers) and includes storage resources, a unique network IP, and options to control how the container should run. In a typical configuration, Kubernetes pod 602 runs a main container 604, which must follow the three-step creation process described above. The main container 604 is used to perform tasks for the workload. As mentioned above, the 2-3 seconds required for container creation can cause delays in task execution, especially for tasks that do not require a long execution time. In one or more embodiments of the present invention, a Kubernetes pod 606 is proposed to be created to run a workload. In this case, the main container operates as a sidecar container 608. The sidecar container 608 prevents pod 606 from terminating and transmitting results, for example by running a lightweight loop. Furthermore, one or more of the temporary containers 610 are created in pod 606. The workload and associated tasks can run in these temporary containers 610, and the sidecar container 608 prevents the pod from terminating. Using the sidecar container 608 avoids the need to initialize and sandbox a new container for each task, reducing workload execution time.

[0067] In one or more embodiments of the present invention, a Kubernetes pod 606 can be managed by a pod management module that tracks resource metrics for ephemeral containers. Currently, it is prohibited to set resources on ephemeral containers. However, in one or more embodiments, the pod manager can dynamically create ephemeral containers by imposing resource limits to prevent a particular container from occupying resources beyond the limits of the pod resources. Figure 7 shows a block diagram of a pod manager performing pod resource management according to one or more embodiments of the present invention. In one or more embodiments, the pod manager 702 can retrieve or receive resource utilization metrics from the resource metrics API 730, or both. The pod manager 702 can calculate the number of containers that can be started, which can be obtained based on an analysis of available resources with respect to the workload or user-defined metrics or a combination thereof. The pod manager 702 then creates and terminates a container with Cgroup 710 to which resources are allocated. These steps can be repeated as many times as necessary to run the workload. The pod manager 702 interacts with the container runtime interface (CRI) 704 and the container runtime engine 705 to dynamically create and terminate container 720. The sidecar container 718, as previously mentioned, for example, runs a lightweight loop to prevent pod 706 from running. Container 720 can be built using the namespace 708 within the pod. The Linux® kernel has a feature called “namespaces”. Linux® kernel namespaces are the main building blocks of Linux® containers, isolating applications into different “user spaces” such as networks, processes, users, and file systems. Namespaces isolate and virtualize system resources for process assemblies. Some examples of resources that can be virtualized include process IDs, hostnames, and user IDs. Namespaces usually refer to namespace types, and may also refer to specific instances of those types.The Linux® operating system is initialized with a single instance for each namespace type. Additional namespaces may be created or merged after initialization. The Linux® kernel also provides a feature called control groups, also known as "cgroups," which allow for the limiting and prioritization of resources (such as CPU, memory, block I / O, and network). The cgroup feature limits various host resources, including CPU count and usage, disk performance, memory, and other process limits.

[0068] In one or more embodiments of the present invention, ephemeral containers can be created and terminated within a pod based on resource allocation, workload, or a combination thereof. Furthermore, in one or more embodiments of the present invention, the pod can be scaled up, scaled down, or both. As the number of ephemeral containers within the pod increases, the pod resource approaches a limit where it can no longer create any more ephemeral containers within the pod. In this case, a horizontal pod autoscaler (HPA) can be used to scale up, scale down, or both of the pod. Figure 8 shows a block diagram of a pod manager and a horizontal pod autoscaler according to one or more embodiments of the present invention. The pod manager 802 can receive the workload 850 again and, using resource metric information collected from the resource metric API 830, create a pod 810 having a sidecar container 814 and one or more ephemeral containers 812. The pod manager 802 can trigger the HPA 806 to scale up or scale down the pod by updating the custom metric API 804 and configuration based on the resource, workload queue, or a combination thereof. Therefore, more pods 810 can be created to run workload 850. Furthermore, when the workload queue is smaller (below a threshold), the HPA 806 can scale down, terminating one or more pods 810 to save resources. In some embodiments, workload 850 may be received from a workload queue manager. The replication controller 808 can scale pods by creating multiple pods 810. This is referred to as "replication". The replicated pods may be created and managed as a group by an abstraction called a controller.

[0069] In one or more embodiments of the present invention, it is possible to scale up or down or both of multiple pods. As multiple pods are created, a controller module is required to monitor the workload and communicate with the pod manager and workload queue manager to maintain the workload. Figure 9 shows a block diagram of a controller module communicating with the pod manager and workload queue according to one or more embodiments of the present invention. In one or more embodiments, the controller 904 can communicate with the pod manager 902, which creates a pod 910 having a sidecar container 914 and one or more temporary containers 912. The controller 904 also communicates with the queue manager 906, which manages the workload queue 908. In one or more embodiments, the workload queue 908 may be a Message Queuing Telemetry Transport (MQTT) queue. The MQTT queue is illustrative and is not intended to limit the types of queues used herein. In one or more embodiments of the present invention, the controller 904 retrieves or receives task definitions from an API server or both. The controller 904 can place tasks into the queue 908 via the queue manager 906. The controller 904 then triggers the pod manager 902 to create a temporary container 912 and a sidecar container 914. The sidecar container 914 monitors the temporary container 912 and sends the results to the message queue 908. The controller 904 retrieves the results from the message queue 908 and updates the task accordingly.

[0070] Figure 10 shows a system for dynamically scaling workload execution in a cloud environment according to one or more embodiments of the present invention. System 1000 includes many of the components described in Figures 6-9. Thus, system 1000 includes a pod manager 1002, a controller 1004, a queue manager 1006, a resource metrics API 1008, an HPA 1010, a replication controller 1012, a container runtime interface 1014, and a workload / message queue 116. One or more tasks 1018 may come in from the API server 1024 and be sent to the controller 1004. The controller 1004 places the tasks into queue 1016 through the queue manager 1006. The controller 1004 can trigger the pod manager 1002 to create a temporary container 1034 in a pod 1032, as well as a sidecar container 1036 for monitoring and sending results to queue 1016. The resource metrics API 1008 monitors the resource metrics of pod 1032. The HPA 1010 and replication controller 1012 can add, terminate, or bother pods based on tasks, resources, workloads, or a combination thereof. The pod manager 1002 manages each pod 1032 and can add, terminate, or bother temporary containers 1034 based on resource allocation metrics.

[0071] In one or more embodiments, system 1000 also includes Kubernetes components such as Etcd 1022 and node component kubelet 1026. Etcd is a persistent, lightweight, distributed key-value datastore that reliably stores configuration data for a Kubernetes cluster representing the overall state of the cluster at any given point in time. kubelet is an agent that verifies that a container is running within a pod. kubelet ensures that a container specified in a set of PodSpecs provided through various mechanisms is running and healthy.

[0072] Figure 11 shows a flowchart of a method for dynamically scaling workload execution in a cloud environment according to one or more embodiments of the present invention. At least part of Method 1100 can be executed by one or more servers, such as a public cloud 120 or a private cloud 130 or a combination thereof, as shown in Figure 1. Method 1100 includes the step of a controller receiving a workload having one or more tasks, as shown in block 1102. In block 1104, Method 1100 includes the step of generating a first pod having a first sidecar container. The first pod is, for example, a Kubernetes pod. In block 1106, Method 1100 includes the step of generating one or more ephemeral containers for the first pod based on the workload and one or more resource allocation metrics for the pod. In block 1108, Method 1100 may include the step of executing one or more tasks within one or more ephemeral containers. The sidecar containers, for example, run lightweight loops to prevent the pod from terminating. In block 1110, method 1100 includes a step of monitoring one or more resource allocation metrics for a pod. The resource allocation metrics may be obtained from a resource monitoring API. Then, in block 1112, method 1100 includes a step of generating at least one new temporary container within the first pod based on one or more resource allocation metrics and workload for the pod.

[0073] Additional processes may also be included. The processes shown in Figure 11 are illustrative, and other processes may be added, or existing processes removed, modified, or reorganized without departing from the scope and intent of this disclosure.

[0074] In this specification, various embodiments of the present invention are described with reference to the relevant drawings. Alternative embodiments of the present invention can be devised without departing from the scope of the invention. Various connections and positional relationships (e.g., above, below, adjacent, etc.) are described between elements in the following description and drawings. These connections or positional relationships, or both, may be direct or indirect unless otherwise specified, and the invention is not intended to be limited thereto. Thus, the connection between entities may refer to either a direct or indirect connection, and the positional relationship between entities may refer to either a direct or indirect positional relationship. Furthermore, the various tasks and process steps described herein can be incorporated into a more comprehensive procedure or process having additional steps or functions not described in detail herein.

[0075] One or more of the methods described herein can be implemented using any or a combination of the following technologies: discrete logic circuits having logic gates for implementing logic functions for data signals, application-specific integrated circuits (ASICs) having appropriate combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and each of these technologies is well known in the art.

[0076] For the sake of brevity, prior art relating to the manufacture and use of embodiments of the present invention may or may not be described in detail herein. Specifically, various embodiments of computing systems and particular computer programs for implementing the various technical features described herein are well known. Therefore, for the sake of brevity, many prior implementation details are mentioned only briefly herein, or are omitted entirely without providing details of well known systems or processes or combinations thereof.

[0077] In some embodiments, various functions or operations may be performed at a given location, in connection with the operation of one or more devices or systems, or both. In some embodiments, some of the given functions or operations may be performed at a first device or location, and the remaining functions or operations may be performed at one or more additional devices or locations.

[0078] The terms used herein are intended solely to describe specific embodiments and are not intended to limit them. Where used herein, singular nouns such as “a,” “an,” and “the” are intended to include plural nouns unless otherwise specified in the context. Furthermore, where used herein, the terms “comprises” or “comprising” or both specify the presence of an expressed feature, integer, stage, operation, element or component, or a combination thereof, but it will be understood that this does not exclude the presence or addition of one or more other features, integers, stages, operations, element components, or groups or combinations thereof.

[0079] In the following claims, the corresponding structures, materials, actions, and equivalents of all means-plus-function elements or step-plus-function elements are intended to include any structures, materials, or actions for performing a function in combination with any other specifically claimed elements. While this disclosure is presented for illustrative and explanatory purposes, it is not intended to be exhaustive or to limit the disclosed forms. Many modifications and changes will become apparent to those skilled in the art without departing from the scope and spirit of this disclosure. The embodiments have been selected and described to best illustrate the principles and practical applications of this disclosure and to enable those skilled in the art to understand the disclosure in terms of various embodiments with various modifications suitable for specific intended uses.

[0080] The diagrams shown herein are illustrative. Many modifications can be made to the diagrams or the stages (or actions) described herein without departing from the spirit of this disclosure. For example, actions can be performed in a different order, and actions can be added, deleted, or modified. Also, the term “coupled” indicates that there is a signal path between two elements, and does not imply a direct connection between elements without an intervening element / connection. All of these variations are considered part of this disclosure.

[0081] The following definitions and abbreviations are for use in interpreting the claims and specification. Where used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains,” or “containing,” or other variations thereof, are intended to cover non-exclusive inclusion. For example, a component, mixture, process, method, article, or apparatus containing a list of elements is not necessarily limited to these elements and may include other elements not expressly enumerated or that are specific to such component, mixture, process, method, article, or apparatus.

[0082] Furthermore, the term “exemplary” is used herein to mean “to serve as an example, case, or illustration.” No embodiment or design described herein as “exemplary” is necessarily construed to be preferable or advantageous to any other embodiment or design. The terms “at least one” and “one or more” are understood to include any integer greater than or equal to 1, i.e., 1, 2, 3, 4, etc. The term “multiple” is understood to include any integer greater than or equal to 2, i.e., 2, 3, 4, 5, etc. The term “connection” may include both indirect and direct “connections.”

[0083] The terms “about,” “substantially,” and “approximately,” and their variations, are intended to include the degree of error associated with the measurement of a particular quantity based on equipment available at the time of filing this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0084] The present invention may be a system, method, or computer program product, or a combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or more mediums) having computer-readable program instructions that cause a processor to implement aspects of the present invention.

[0085] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any preferred combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved raised structures on which instructions are recorded, and any preferred combination of those described above. When used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers these computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0087] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® and C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuit in order to perform aspects of the present invention.

[0088] This specification describes aspects of the invention with reference to flowcharts, block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and combinations of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0089] By providing these computer-readable program instructions to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, a machine may be generated such that instructions executed via the processor of the computer or other programmable data processing device create means for implementing functions / operations specified in one or more blocks in a flowchart or block diagram, or both. Alternatively, by storing these computer-readable program instructions in a computer-readable storage medium capable of instructing a computer, a programmable data processing device, or other device, or a combination thereof, to function in a particular manner, the computer-readable storage medium containing the instructions may constitute a product containing instructions that implement modes of functions / operations specified in one or more blocks in a flowchart or block diagram, or both.

[0090] Alternatively, computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device, and a series of operational steps may be executed on the computer, other programmable device, or other device to generate a computer implementation process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified by one or more blocks in a flowchart or block diagram or both.

[0091] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown within a block may be performed in an order other than that shown in the figure. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may be executed in reverse order depending on the functions involved. It will also be seen that each block in a block diagram or flowchart, or a combination thereof, and combinations of blocks in a block diagram or flowchart, or a combination thereof, can be implemented by a dedicated hardware-based system that performs a specified function or operation, or can implement a combination of dedicated hardware and computer instructions.

[0092] While various embodiments of the present invention have been presented for illustrative purposes, they are not intended to be exhaustive or limitless. Those skilled in the art will see many modifications and changes without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best describe the principles, practical applications, or technical improvements to the technologies available on the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. The controller receives a workload having one or more tasks, A step of generating a first pod having a first sidecar container, A step of generating one or more temporary containers for the first pod based on the workload and one or more resource allocation metrics for the first pod, The steps include: executing the one or more tasks within the one or more temporary containers; A step of monitoring the one or more resource allocation metrics for the first pod, If the number of temporary containers in the first pod has not reached the maximum number of temporary containers in the first pod, which is dynamically calculated based on the one or more resource allocation metrics for the first pod, then the steps include generating at least one new temporary container in the first pod based on the workload. A computer implementation method comprising the following:

2. A step of terminating at least one of the one or more temporary containers in the first pod based on the one or more resource allocation metrics. The computer implementation method according to claim 1, further comprising:

3. A step of determining the maximum number of temporary containers in the first pod based on the one or more resource allocation metrics, If the workload requires a number of temporary containers exceeding the maximum number of temporary containers in the first pod, the steps include generating a second pod having one or more second temporary containers. The computer implementation method according to claim 1 or 2, further comprising:

4. The computer implementation method according to claim 1 or 2, wherein the one or more tasks are stored in a queue.

5. The computer implementation method according to claim 4, wherein the queue includes a message queuing telemetry transport queue.

6. The computer implementation method according to claim 1 or 2, wherein the first pod includes a Kubernetes pod.

7. The computer implementation method according to claim 1 or 2, wherein the first sidecar container performs a loop to prevent the first pod from terminating.

8. Memory with computer-readable instructions, The system comprises one or more processors for executing the computer-readable instructions, wherein the computer-readable instructions control the one or more processors, Procedures for receiving workloads containing one or more tasks, A procedure for generating a first pod including a first sidecar container, A procedure for generating one or more temporary containers for the first pod based on the workload and one or more resource allocation metrics for the first pod, A procedure for performing the one or more tasks within the one or more temporary containers, A procedure for monitoring the one or more resource allocation metrics for the first pod, If the number of temporary containers in the first pod has not reached the maximum number of temporary containers in the first pod, which is dynamically calculated based on the one or more resource allocation metrics for the first pod, then the procedure involves generating at least one new temporary container in the first pod based on the workload. A system that performs actions including those mentioned above.

9. The aforementioned operation, A procedure for terminating at least one of the one or more temporary containers in the first pod based on the one or more resource allocation metrics. The system according to claim 8, further comprising:

10. The aforementioned operation, A procedure for determining the maximum number of temporary containers in the first pod based on the one or more resource allocation metrics, If the workload requires a number of temporary containers exceeding the maximum number of temporary containers in the first pod, the procedure involves generating a second pod containing one or more second temporary containers. The system according to claim 8 or 9, further comprising:

11. The system according to claim 8 or 9, wherein one or more of the tasks are stored in a queue.

12. The system according to claim 11, wherein the queue includes a message queuing telemetry transport queue.

13. The system according to claim 8 or 9, wherein the first pod includes a Kubernetes pod.

14. The system according to claim 8 or 9, wherein the first sidecar container performs a loop to prevent the first pod from ending.

15. In the processor, The controller has a procedure for receiving a workload that includes one or more tasks, A procedure for generating a first pod including a first sidecar container, A procedure for generating one or more temporary containers for the first pod based on the workload and one or more resource allocation metrics for the first pod, A procedure for performing the one or more tasks within the one or more temporary containers, A procedure for monitoring the one or more resource allocation metrics for the first pod, If the number of temporary containers in the first pod has not reached the maximum number of temporary containers in the first pod, which is dynamically calculated based on the one or more resource allocation metrics for the first pod, then the procedure involves generating at least one new temporary container in the first pod based on the workload. A computer program designed to execute something.

16. The aforementioned processor, A procedure for terminating at least one of the one or more temporary containers in the first pod based on the one or more resource allocation metrics. The computer program according to claim 15, which further performs the following.

17. The aforementioned processor, A procedure for determining the maximum number of temporary containers in the first pod based on the one or more resource allocation metrics, If the workload requires a number of temporary containers exceeding the maximum number of temporary containers in the first pod, the procedure involves generating a second pod containing one or more second temporary containers. A computer program according to claim 15 or 16, which further causes the following to be executed.

18. The computer program according to claim 15 or 16, wherein one or more of the aforementioned tasks are stored in a queue.

19. The computer program according to claim 18, wherein the queue includes a message queuing telemetry transport queue.

20. The computer program according to claim 15 or 16, wherein the first pod includes a Kubernetes pod, and the first sidecar container performs a loop to prevent the first pod from terminating.

Images