Predictive autoscaling and resource optimization

Abstract

Techniques for predictive autoscaling and resource optimization for software deployment. In one implementation, the user declares performance objectives, and machine learning is used to determine the lowest-cost resource configuration that satisfies these objectives, based on application behavior and load profiles. In another implementation, deployment convergence is monitored, and relevant feedback is provided to improve predictions, behavioral modeling, and resource estimation over time.

Description

Background Technology

[0001] Capacity planning and cost optimization for software operations are areas of ongoing research and development. Overprovisioning leads to resource waste and additional costs, but the industry standard averages overprovisioning at 80-93%. Underprovisioning leads to performance degradation and SLA violations. Research shows that performance degradation in web applications can cause churn to increase by up to 75%. "Preliminary results on cloud service availability… show an average of 7,738 hours of unavailability or 99.91% availability per year…based on industry-accepted hourly costs, these outages cost a total of approximately $285 million." (Cérin et al., Downtime Statistics for Current Cloud Solutions, Updated March 2014).

[0002] Manually determining capacity is practically always flawed due to the dynamic nature of resource utilization and application load. Reactive autoscaling, by definition, cannot meet loads in advance. Threshold-based autoscaling requires significant effort and fails to align with defined service-level objectives, even when using custom application-level metrics. A utilization threshold of up to 80% can result in 20% capacity undercapacity. Research indicates that threshold-based autoscalers cannot adapt to changing workloads.

[0003] Techniques to address these and other shortcomings associated with capacity planning and cost optimization are desirable. Summary of the Invention

[0004] A cost and performance management scheme for software resource resourcing and scaling is disclosed. In a specific implementation, the system runs on a software deployment orchestration platform (e.g., Kubernetes) that exposes application and resource metrics and provides standard scaling and resource resourcing mechanisms. Users declare performance objectives, and the system learns application behavior and load profiles to determine the lowest-cost resource configuration to meet those declared performance objectives. Attached Figure Description

[0005] Figure 1 A diagram depicts examples of predictive autoscaling and resource optimization systems.

[0006] Figure 2 A graph depicts a comparison of resource supply based on recommendations from a reactive recommendation engine with resource supply based on recommendations from a predictive recommendation engine.

[0007] Figure 3 It is a graph comparing the predicted total load with the actual total load, and a graph displayed by the associated code.

[0008] Figure 4A flowchart depicts an example of a predictive autoscaling and resource optimization approach.

[0009] Figure 5 A flowchart depicts an example of combining machine learning processes to generate predictive autoscaling and resource optimization results.

[0010] Figure 6 A diagram depicts an example of a system used to generate the lowest-cost optimization parameters. Detailed Implementation

[0011] Figure 1 Figure 100 illustrates an example of a predictive auto-scaling and resource optimization system. As used herein, resources can be characterized as a central processing unit (CPU), memory, network input / output (I / O), disk I / O, graphics processing unit (GPU), and / or other applicable resources. Figure 100 includes a computer-readable medium (CRM) 102, a service level agreement (SLA) metrics data store 104 coupled to the CRM 102, a feedforward control system 106 coupled to the CRM 102 for a software orchestration platform, a convergent deployment 122 coupled to the CRM 102, and a load distribution and metrics engine 124 coupled to the CRM 102. The feedforward control system 106 for the software orchestration platform includes a declarative performance interface engine 108, a predictive autoscaling and resource optimization operator engine 110, a dynamic estimation engine 112, an application load prediction engine 114, a minimum cost optimization engine 116, a scaling resource optimal configuration executor engine 118, and a convergence deployment, resource, and application-level metric collection engine 120. The predictive autoscaling and resource optimization system can be fully implemented; implemented as a phased integration (e.g., integration into a Kubernetes cluster), where customers control whether changes are made in real-time and how many changes are allowed; or implemented using platform data samples to provide cost savings and / or performance improvement reports.

[0012] CRM 102 is intended to represent a computer system or a network of computer systems. As used herein, "computer system" can include or be implemented as a dedicated computer system for performing the functions described herein. Typically, a computer system includes a processor, memory, non-volatile storage, and interfaces. A typical computer system typically includes at least a processor, memory, and devices (e.g., buses) that couple the memory to the processor. The processor can be, for example, a general-purpose central processing unit (CPU) (e.g., a microprocessor) or a dedicated processor (e.g., a microcontroller).

[0013] For example, but not limited to, the memory of a computer system includes random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). Memory can be local, remote, or distributed. Non-volatile memory is typically a magnetic floppy disk or hard disk, magneto-optical disk, optical disk, read-only memory (ROM) (e.g., CD-ROM, EPROM, or EEPROM), magnetic card or optical card, or another form of storage for large amounts of data. During software execution, some of this data is typically written into memory via a direct memory access process through a bus coupled to the non-volatile memory. Non-volatile memory can be local, remote, or distributed, but it is optional because the system can be created using all applicable data available in memory.

[0014] Software in a computer system is typically stored in non-volatile memory. In fact, for large programs, it may not even be possible to store the entire program in memory. To enable software to run, it is moved, if necessary, to a computer-readable location suitable for processing, and for illustrative purposes herein, this location is referred to as memory. Even when software is moved to memory for execution, the processor typically utilizes hardware registers to store values ​​associated with the software, as well as local caches, ideally for speeding up execution. As used herein, when a software program is referred to as "implemented in a computer-readable storage medium," it is assumed that the software program is stored in an applicable, known, or convenient location (from non-volatile memory to hardware registers). A processor is considered "configured to execute the program" when at least one value associated with the program is stored in a processor-readable register.

[0015] In one operational example, a computer system can be controlled by operating system software, which is a software program that includes a file management system, such as a disk operating system. An example of operating system software with associated file management system software is the Windows family of operating systems from Microsoft Corporation in Redmond, Washington, and its associated file management system. Another example of operating system software with associated file management system software is the Linux operating system and its associated file management system. File management systems are typically stored in non-volatile memory and enable the processor to perform various actions required by the operating system to input and output data and store data in memory, including storing files on non-volatile memory.

[0016] A computer system's bus couples the processor to an interface. The interface facilitates the coupling of devices and the computer system. The interface can be used for input and / or output (I / O) devices, modems, or networks. For example, but not limited to, I / O devices can include keyboards, mice, or other pointing devices, disk drives, printers, scanners, and other I / O devices, including display devices. For example, but not limited to, display devices can include cathode ray tube (CRT), liquid crystal displays (LCDs), or some other suitable, known, or convenient display devices. For example, but not limited to, modems can include analog modems, IDSN modems, cable modems, and other modems. For example, but not limited to, network interfaces can include token ring interfaces, satellite transmission interfaces (e.g., "direct PC"), or other network interfaces used to couple a first computer system to a second computer system. An interface can be considered part of a device or computer system.

[0017] Computer systems may be compatible with, be part of, or be implemented through cloud-based computing systems. As used herein, a cloud-based computing system is a system that provides virtualized computing resources, software, and / or information to client devices. Computing resources, software, and / or information can be virtualized by maintaining centralized services and resources accessible to edge devices through communication interfaces such as networks. "Cloud" can be a marketing term and, for the purposes of this document, can include any network described herein. Cloud-based computing systems may involve service subscriptions or utility pricing models. Users may access cloud-based computing systems through a web browser or other container applications located on their client devices.

[0018] A computer system may be implemented as an engine, as part of an engine, or through multiple engines. As used herein, an engine comprises at least two components: 1) a dedicated or shared processor or a portion thereof; and 2) hardware, firmware, and / or software modules executed by the processor. A portion of one or more processors may include some portion of hardware less than all the hardware comprising any of the specified one or more processors, such as a subset of registers, a portion of the processor dedicated to one or more threads of a multithreaded processor, a time slice of the processor entirely or partially dedicated to performing a portion of the engine's functionality, etc. Thus, a first engine and a second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on implementation-specific considerations or other considerations, an engine may be centralized or its functionality may be distributed. An engine may include hardware, firmware, or software contained in a computer-readable medium for execution by a processor. The processor uses implemented data structures and methods to transform data into new data, such as as described with reference to the accompanying figures.

[0019] The engine described herein, or the engine that can implement the systems and devices described herein, can be a cloud-based engine. As used herein, a cloud-based engine is an engine that can use a cloud-based computing system to run applications and / or functions. All or part of the applications and / or functions can be distributed across multiple computing devices and are not necessarily limited to a single computing device. In some implementations, a cloud-based engine can execute functions and / or modules accessed by end users through a web browser or container application without requiring that function and / or module to be installed locally on the end user's computing device.

[0020] As used herein, a data storage area is intended to include a repository with any applicable data organization, including tables, comma-separated value (CSV) files, traditional databases (such as SQL), or other applicable known or convenient organization formats. For example, a data storage area can be implemented as software contained in a physical computer-readable medium on a general-purpose or special-purpose machine, in firmware, in hardware, in a combination thereof, or in an applicable known or convenient device or system. While the physical location and other characteristics of the components associated with a data storage area are not essential to understanding the techniques described herein, components associated with a data storage area (such as a database interface) can be considered as “part” of the data storage area, part of certain other system components, or a combination thereof.

[0021] Data storage can include data structures. As used herein, a data structure is associated with a specific way of storing and organizing data in a computer so that it can be used effectively within a given context. Data structures are typically based on the computer's ability to access and store data specified by an address (i.e., a bit string that can be stored in memory and manipulated by a program) at any location in its memory. Thus, some data structures are based on calculating the address of a data item using arithmetic operations; while others are based on storing the address of a data item within the structure itself. Many data structures use these two principles, sometimes combined in a non-trivial way. Implementing a data structure typically requires writing a set of procedures for creating and manipulating instances of that structure. The data storage described in this article can be a cloud-based data storage. A cloud-based data storage is a data storage system compatible with cloud-based computing systems and engines.

[0022] Assuming CRM includes a network, this network can be a suitable communications network, such as the Internet or an infrastructure network. The term "Internet" as used herein refers to a network that uses certain protocols (such as TCP / IP) and possibly others (such as Hypertext Transfer Protocol (HTTP) used to construct Hypertext Markup Language (HTML) documents for the World Wide Web ("the Network"). More generally, a network can include, for example, a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a Campus Network (CAN), or a Local Area Network (LAN), but a network can be at least theoretically of a suitable size or characterized in some other way (to name just a few alternatives, such as a Personal Area Network (PAN) or a Home Area Network (HAN)). Networks can include enterprise-specific networks and Virtual Private Networks (VPCs) (collectively, referred to as private networks). As the name suggests, a private network is controlled by a single entity. A private network can include headquarters and optional regional offices (collectively, referred to as offices). Many offices allow remote users to connect to private network offices via some other network (such as the Internet).

[0023] Refer again Figure 1For example, SLA metrics data storage area 104 is intended to represent a data storage area including a data structure representing a declared performance objective for software deployment. SLA metrics data storage area 104 may include a Service Level Indicator (SLI) data structure. In information technology, SLI is a measure of the level of service provided by a service provider to a customer. SLI forms the basis of Service Level Objectives (SLOs), which in turn form the basis of SLAs. SLA metrics data storage area 104 may alternatively or additionally include an SLO data structure. Therefore, as used herein, one or both of SLI and SLO can be considered SLA metrics. Although it is assumed for illustrative purposes that granular SLI or SLO is converted to a declarative performance objective, combinations of SLI and / or SLO can be formed into an SLA data structure, stored in SLA metrics data storage area 104, and converted into aggregated declarative performance objectives. In a particular implementation, the declared performance objective represents a service consumer at least partially responsible for software deployment. A service consumer may include an entity that builds, provides, or hosts a software product that requires one or more services (e.g., computing resources) to achieve the desired functionality and operation. For example, service consumers may include companies, organizations, institutions, enterprises, groups, individuals, or other applicable entities or groups of entities.

[0024] The feedforward control system 106 for a software orchestration platform is designed to represent a system comprising an engine and data storage area for proactively scaling up in advance to prepare for predicted loads, thereby mitigating provisioning delays in software deployment. Proactive scaling up of applications and application resources enables resource provisioning to meet future loads. In various implementations, the feedforward control system 106 for the software orchestration platform reduces the cost of running applications by decreasing resource consumption; improves application performance by continuously provisioning and scaling up resources for applications so that they can meet the performance objectives of current and predicted loads; gives users confidence in their set SLAs by allowing the resource provisioning and scaling mechanism to exit and satisfy it; and / or reduces the manual time and effort involved in analyzing applications to properly provision and scale up resources. In a particular implementation, the feedforward control system 106 for the software orchestration platform predicts random and regular workloads with up to 90% accuracy; preemptive resource provisioning results in an average reduction of 90% in SLA violations. Preemptive resource provisioning of applications means that undersupply is at least improved and ideally eliminated. In a particular implementation, the feedforward control system 106 used in the software orchestration platform can learn from the decisions made in order to improve forecasting, modeling, resource estimation and other applicable decisions.

[0025] The declarative performance interface engine 108 is designed to represent an engine that coordinates with control system type technology (described in conjunction with other engines of the feedforward control system 106 for the software orchestration platform) to consume target performance metrics associated with software deployment from the SLA metric data store 104 and provide declarative performance targets to the control system type technology to design a startup procedure to achieve those targets. In a particular implementation, the declarative performance interface engine 108 defines its declarations as “declarative performance” and its definition as a reference or target of “declarative performance purpose” or “declarative performance target.” The declarative performance interface engine 108 enables human or artificial agents of service providers (or service consumers) to define SLA metrics (e.g., SLI or SLO) to be stored in the SLA metric data store 104 as targets for resource configuration of applications by the feedforward control system 106 for the software orchestration platform. With declarative performance, human agents do not need to manually analyze, configure, and resource-configure their applications every time the software or application load changes; and the artificial agents do not need to be artificial intelligence. Advantageously, the declarative performance interface engine 108 allows service providers to rely on declared SLOs (or SLOs derived from one or more declared SLIs) for providing SLAs to service consumers. In certain implementations, implementing the declarative performance interface engine 108 and control system type techniques described herein results in an average increase in SLA compliance of 70%.

[0026] The predictive autoscaling and resource optimization operator engine 110 is designed to represent an engine that automatically scales without overprovisioning in response to predicted resource demands. Resource optimization is defined as supplying the minimum resources required to meet the stated performance objectives. In a particular implementation, the predictive autoscaling and resource optimization operator engine 110 automatically makes changes, recommends more cost-effective SLOs, and sends alerts about potential performance degradation. In a particular implementation, the predictive autoscaling and resource optimization operator engine 110 is robust to seasonal and stochastic application load and resource signatures using a deep learning approach sensitive to trends and seasonality, and is trained to be sensitive to leading indicators of stochastic bursts. In a particular implementation, the predictive autoscaling and resource optimization operator engine 110 is easy to install, thus supporting interchangeable metric collectors and load balancers; capable of running in the cloud or locally (on-prem); and can make recommendations in as little as 5 minutes (coordinated with the engine and data storage area of ​​the feedforward control system 106 used for the software orchestration platform).

[0027] In certain implementations, the predictive autoscaling and resource optimization operator engine 110 does not interfere with the Kubernetes scheduler. Kubernetes is an open-source container orchestration system maintained by the Cloud Native Computing Foundation for automating the deployment, scaling, and management of applications. It aims to provide a "platform for automating the deployment, scaling, and operation of application containers across host clusters." It utilizes a suite of container tools, including Docker. Many cloud services offer Kubernetes-based platforms or infrastructure as a service (PaaS or IaaS), on which Kubernetes can be deployed as a platform service. Many vendors also offer their own branded Kubernetes distributions. The paper "The Kubernetes Architectural Roadmap" by Brian Grant, Tim Hockin, and Clayton Colman, last updated April 20, 2017, is incorporated herein by reference.

[0028] In certain implementations, the predictive autoscaling and resource optimization operator engine 110 works well with horizontal autoscalers and cluster autoscalers. For example, the Kubernetes horizontal pod autoscaler automatically scales the number of pods in a replication controller, deployment, or replicaset based on observed CPU utilization. As another example, Oracle Cloud Platform allows server instances to automatically scale inwards or outwards their clusters by defining autoscaling rules based on CPU and / or memory utilization to determine when to add or remove nodes.

[0029] In a particular implementation, the predictive autoscaling and resource optimization operator engine 110 receives the following outputs as input: 1) the output of measurements from the convergence deployment, resource, and application-level metrics collection engine 120 and 2) the output of predictions from the application load prediction engine 114. In another particular implementation, the predictive autoscaling and resource optimization operator engine 110 determines the measurement error for performance purposes, and this measurement error is provided to the dynamic estimation engine 112.

[0030] The dynamic estimation engine 112 is designed to represent an engine that estimates the minimum amount of resources required to meet service level objectives under predicted and current loads. In a particular implementation, the dynamic estimation engine 112 models application behavior in terms of resource utilization and models performance as a response to load. By modeling the application's response and resource utilization under load, estimations of vertical and horizontal autoscaling can be achieved, and based on this modeling, estimates can be made of how many resources will be used under a specific load and the corresponding service indicators (e.g., the time to process requests); deployment requests can be set accordingly.

[0031] Application load prediction engine 114 provides load estimates for future times. Depending on implementation-specific factors, the future time can be configured as short as one minute or as long as one hour. In a particular implementation, application load prediction engine 114 provides 91% accurate load prediction. Advantageously, by predicting incoming application load in advance, it can scale longitudinally before load events occur, even across a wide variety of workloads. For example, using deep learning methods that have proven to generalize across a wide range of workloads, application load prediction engine 114 can predict seasonal, trendy, bursty, and random loads with high accuracy (at least 83% or up to 95%). Based on requests and understanding of the workload, signature limits can be set on deployments according to utilization patterns (e.g., bursty loads result in higher limits, while stable loads result in lower limits).

[0032] The lowest-cost optimization engine 116 uses the current load, the predicted load from the application load prediction engine 114, the application behavior model from the dynamic estimation engine 112, and the declared purpose from the declarative performance interface engine 108 to find the lowest cost for the application modeled for performance that suits the declared purpose. This is a trade-off between application replication (horizontal scaling) and resources (vertical scaling). This focus on performance optimization leads to cost optimization. Advantageously, in certain implementations, this results in an average cost saving of up to 80% compared to systems without this focus.

[0033] The Extended Resource Optimal Configuration Executor Engine 118 is designed to represent executors specific to the problem space. In a particular implementation, the predictive model is robust to different time series profiles because the parametric time series model is tuned to a certain type of time series profile by its nature. These profiles include switching workloads, bursty workloads, workloads with various trends, and workloads with different seasonal components (seconds, minutes, hours, etc.). This can be achieved by training the model offline to make predictions for many different time series profiles; a recurrent neutral network can be used for this purpose. The offline model is then deployed in the system, and the Extended Resource Optimal Configuration Executor Engine 118 can be characterized as specific to the problem space associated with a certain type of time series profile.

[0034] The Extended Resource Optimization Executor Engine 118 performs optimal configuration of extended resources, such as the number of replicas, the size of resource requests, and the quality of service (defined by limitations on application resource usage that may be terminated or suppressed). By understanding the application under load, the number of replicas and virtual machine (VM) instance types selected to meet predicted demand at the lowest cost for performance purposes can be optimized, thereby minimizing the resources required to meet predicted demand.

[0035] In a particular implementation, the extended resource optimal configuration executor engine 118 uses heuristics specific to extended resources (e.g., maximum allocatable resources) and oscillation damping through consensus-based recommendations. The extended resource optimal configuration executor engine 118 enables the application to be executed as a convergent deployment 122.

[0036] The convergence deployment, resource, and application-level metrics collection engine 120 is designed to represent an engine that measures feedback and feedforward (prediction) based on current performance and predicted load. Feedback may appear as system output from the scaling resource optimization executor engine 118 or the convergence deployment 122, or as additional data associated with related convergence deployments of the convergence deployment 122 provided through or observed on the CRM 102. Feedback and feedforward are used by the predictive autoscaling and resource optimization operator engine 110 to adjust recommendations. In a particular implementation, the convergence deployment, resource, and application-level metrics collection engine 120 monitors performance indicators and resource usage, including SLI metrics such as request count and request duration, and resource utilization metrics such as memory, CPU, disk I / O, and network I / O per container and pod.

[0037] Converging Deployment 122 is intended to represent an engine that executes applications with converged configuration. Converging configuration is executed by the scaling resource optimal configuration executor engine 118 to combine predictive auto-scaling and resource optimization configuration.

[0038] The load distribution and metrics engine 124 is intended to represent an engine that specifies how application load metrics are collected and configured for distribution. In a particular implementation, the load distribution and metrics engine 124 performs load balancing on traffic going to (or coming from) the convergence deployment 122. Load balancing improves the distribution of workloads across multiple computing resources, such as computers, computer clusters, network links, CPUs, or disk drives. Load balancing aims to optimize resource utilization, maximize throughput, minimize response time, and avoid overloading any single resource. Using multiple components with load balancing instead of a single component can improve reliability and availability through redundancy. Load balancing typically involves dedicated software or hardware, such as multi-layer switches or Domain Name System (DNS) server processes. Load balancing differs from channel binding in that load balancing divides traffic between network interfaces on a network socket (OSI Layer 4) basis, while channel binding means dividing traffic between lower-level physical interfaces on a data link (OSI Layer 2) basis using protocols such as shortest path bridging per packet. However, channel binding is considered load balancing in this document. In alternative solutions, the agent takes on load distribution and metrics functions, replacing what might be called a "load balancer".

[0039] For the purpose of distinction in this document, "load distribution and metrics engine" is a more general term for the collection and distribution configuration of application load metrics than a load balancer, agent, or other applicable application-specific load distribution and metrics system. In practice, in certain implementations, an application load metric collection and distribution engine is used instead of a load balancer. For example, load distribution and metrics engine 124 can collect metrics from and configure an application agent (such as Envoy, L7 agents, and communication buses designed for service-oriented architectures of large modems). As another example, load distribution and metrics engine 124 can leverage other load systems, such as message queues. Generally, load distribution and metrics engine 124 can be used across different workloads (as an alternative or supplement to network-based workloads that load balancing satisfies).

[0040] The load distribution and metrics engine 124 is notified by the extended resource optimal configuration executor engine 118 to balance traffic in a manner suitable for convergence deployment 122. The convergence deployment, resource, and application-level metrics collection engine 120 can also collect data from the load distribution and metrics engine 124.

[0041] In the operational example, a human or artificial agent of the service provider (or service consumer) uses the declarative performance interface engine 108 to store SLA metrics, such as SLI or SLO, in the SLA metrics data store 104. Alternatively, the agent can store SLA metrics in the SLA metrics data store 104 via an SLA metrics data store interface (not shown). Depending on what is stored, the SLA metrics data store 104 may be referred to as an SLI data store, an SLO data store, or an SLA data store.

[0042] Continuing this operational example, the declarative performance interface engine 108 transforms data structures into declarative performance objectives for consumption by the predictive auto-scaling and resource optimization operator engine 110. When applicable data becomes available from the convergence deployment, resource, and application-level metrics collection engine 120, the dynamic estimation engine 112 models application behavior in terms of resource utilization and models performance as a response to load, while the application load prediction engine 114 provides load estimates for future times.

[0043] Continuing this operational example, the minimum cost optimization engine 116 uses the predicted load from the application load prediction engine 114, the application behavior model from the dynamic estimation engine 112, and the declared purpose from the declarative performance interface engine 108 to find the minimum cost for the application modeled for performance that aligns with the declared purpose. The predictive autoscaling and resource optimization operator engine 110 provides minimum cost optimization parameters to the scaling resource optimal configuration executor engine 118, which performs convergent deployment 122 and configures the load distribution and metrics engine 124 according to the minimum cost optimization parameters. In a particular implementation, configuring the load distribution and metrics engine 124 involves making the supplied resources known to the load distribution and metrics engine 124, which can happen quite naturally.

[0044] Continuing this operational example, the convergence deployment, resource, and application-level metrics collection engine 120 monitors channels and other resources associated with the convergence deployment 122. These can be processed (to generate, for example, measured outputs) and provided as feedback to the predictive autoscaling and resource optimization operator engine 110, the dynamic estimation engine 112, and the application load prediction engine 114. This feedback can be used to provide an initial dataset or to improve modeling and recommendations over time.

[0045] Figure 2A graph 200 depicts a comparison of resource supply based on the recommendations of a reactive recommendation engine and resource supply based on the recommendations of a predictive recommendation engine. Graph 200 includes a resource consumption curve 202, a predictive supply curve 204, a reactive supply curve 206, a performance degradation region 208, and a waste cost region 210. Resource consumption curve 202 is intended to represent the amount of resources used over time (x-axis) (y-axis). Predictive supply curve 204 is intended to represent resources supplied according to the recommendations of the predictive recommendation engine. Reactive supply curve 206 is intended to represent resources allocated according to the recommendations of a reactive recommendation engine, which is an alternative to the predictive recommendation engine.

[0046] As shown in Table 200, the performance degradation region 208 of the reactive supply curve 206 is larger than that of the predictive supply curve 204. In fact, the predictive supply curve 204 matches or slightly exceeds the resource consumption curve 202 during the measured time period, meaning that the system using the predictive recommendation engine does not experience performance degradation. It can be noted that when the supply curve is smaller than the resource consumption curve 202, performance degradation occurs, indicating that a supply shortage has occurred.

[0047] As shown in Figure 200, the waste cost 210 of the reactive supply curve 206 is greater than that of the predictive supply curve 204. Although the predictive supply curve 204 exceeds the resource consumption curve 202 at most points in Figure 200, the amount of waste cost is significantly less than that associated with the reactive supply curve 206. The reactive supply system cannot achieve correct supply within 5 minutes of responding to the load because, although there are vertical scaling events before 5 minutes (e.g., a 1-2 minute response time), following the curve downwards is difficult and the reactive algorithm degrades over time. In a particular implementation, correct supply (with provisioning insurance) takes less than 5 minutes after the load. Because the reactive supply system cannot achieve correct supply before it receives metrics, calculates requests, and initiates them, it is impossible for the reactive system to act within one minute of the load, which is entirely within the capabilities of the particular implementation. Advantageously, in a particular implementation of this correctly configured predictive system, the correct supply can be achieved x minutes before the load or the resources needed or consumed, where x is a configurable look-ahead time greater than 1 minute and less than 1 hour.

[0048] Wasteful costs occur when the supply curve exceeds the resource consumption curve 202 plus supply insurance. Simply put, the wasteful cost of consuming less than x% of resources within one minute can be termed supply insurance, which is desirable in many cases to ensure that undersupply does not occur. Supply insurance can be defined as the probability that the resource value is less than x% of the supply, or as a 95% probability of the peak-to-average ratio (peak-to-average ratio) and / or peak-to-average ratio (peak factor). In certain implementations, both heuristics are used. The 95% probability of the resource value being less than the supply is used for requests (e.g., how much resource to request), while the peak factor is used to determine limits (e.g., the amount of resource allowed to be consumed beyond the requested amount before termination, suppression, or compression of resource usage). The difference between requests and limits in a software orchestration platform can be termed Quality of Service (QoS), which defines whether resource availability is always guaranteed (i.e., requests and limits are the same) or whether software is allowed to exceed its requests as needed when resources are available (i.e., limits exceed requests).

[0049] Figure 3 Chart 302 compares the predicted total load to the actual total load, and chart 300 shows the associated code display 304. Chart 302, comparing the predicted total load to the actual total load, has an x-axis with timestamps in seconds and a y-axis with load counts per second. It can be seen that the predicted load curve always exceeds the request count curve by a relatively small margin (supply margin). The associated code display 304 indicates that resources include limits and requests, which have been described in the preceding paragraphs.

[0050] Figure 4 Flowchart 400 illustrates an example of a predictive autoscaling and resource optimization approach. Flowchart 400 begins with module 402, which transforms the SLA metric data structure into a declarative performance-oriented structure. The SLA metric data structure can be stored in an SLA metric data store, for example, in conjunction with... Figure 1 The SLA metrics data storage area 104 is described. A declarative performance interface engine (e.g., combined with...) Figure 1 The declarative performance interface engine (108) described can convert SLA metric data structures into declarative performance objects.

[0051] Flowchart 400 continues to module 404, which estimates load forecasts for future times. For new deployments, it should be noted that load forecasting can have limited use because it is essentially just a guess based on known data about the deployment, without the benefit of feedback related to resource utilization and post-deployment performance. Receiving and processing such feedback typically takes several minutes, at which point the load forecast can become a more predictive estimate. Therefore, module 404 can be skipped if not needed until the data becomes useful for making accurate forecasts. Application load forecasting engines (e.g., combined with...) Figure 1The described application load forecasting engine (114) can estimate load forecasts for future times.

[0052] Flowchart 400 continues to module 406, which uses load forecasting and declarative performance objectives to generate minimum cost optimization parameters. As mentioned in the previous paragraph, forecasting may have limited use in accurately generating minimum cost optimization parameters. Furthermore, while model estimates can be provided instead of performance models and application behavior models generated in response to deployment-related feedback, the value of such models is limited. Upon receiving feedback, module 406 can also use the performance model and application behavior model to generate minimum cost optimization parameters. (See the description below in conjunction with modules 416 and 418.) Minimum cost optimization engine (e.g., combined with...) Figure 1 The described minimum cost optimization engine 116 can generate minimum cost optimization parameters using load forecasting, performance models (if applicable), application behavior models (if applicable), and declarative performance objectives.

[0053] Flowchart 400 continues to module 408, which performs convergent deployment based on lowest-cost optimization parameters. This involves optimizing the executor engine with extended resources (e.g., combining...). Figure 1 The described extended resource optimal configuration executor engine 118 can perform converged deployment with optimized parameters at the lowest cost.

[0054] Flowchart 400 continues in parallel to module 410, which configures the load distribution and metrics engine based on the lowest cost optimization parameters. While any applicable module can be configured to execute in parallel with another module, modules 408 and 410 are more likely to execute in parallel and are therefore illustrated quite explicitly. Of course, the modules can be rearranged for serial processing. Expanding resource allocation optimizes the executor engine configuration (e.g., combining...). Figure 1 The described extended resource optimal configuration executor engine 118) can configure load distribution and metrics engine based on parameters that optimize the configuration at the lowest cost.

[0055] Following modules 408 and 410, flowchart 400 continues to module 412, which monitors resources associated with converged deployment. The converged deployment and resource metrics collection engine (e.g., combined with...) Figure 1 The described convergence deployment, resource, and application-level metrics collection engine (120) can monitor resources (including channels) associated with convergence deployments.

[0056] Flowchart 400 continues to module 414, which provides feedback associated with converged deployment. This includes a converged deployment and resource metrics collection engine (e.g., combined with...). Figure 1 The described convergence deployment, resource, and application-level metrics collection engine (120) can provide feedback related to convergence deployment.

[0057] Flowchart 400 returns to module 404 and continues as previously described, and also (in parallel) to module 416, which models application behavior in terms of resource utilization. In a particular implementation, the total time required for modeling application behavior is up to approximately 5 minutes to receive, process, and perform machine learning on the feedback from module 414. Therefore, in Figure 4 In this example description, although an "alternative" model can be used, the introduction of module 416 is not as fast as module 404. Furthermore, flowchart 400 can loop through other modules multiple times before module 416 completes. Dynamic estimation engines (e.g., combined with...) Figure 1 The described dynamic estimation engine 112) can model application behavior in terms of resource utilization. From module 416, flowchart 400 returns to module 406 and continues as previously described.

[0058] Flowchart 400 continues from module 414 to module 418, which models performance as a response to load. In a particular implementation, modeling application behavior requires a maximum of approximately 5 minutes to receive, process, and perform machine learning on the feedback from module 414. Therefore, in Figure 4 In this example description, although an "alternative" model can be used, the introduction of module 418 is not as fast as module 404. Furthermore, flowchart 400 can loop through other modules multiple times before module 418 completes. Dynamic estimation engines (e.g., combined with...) Figure 1 The described dynamic estimation engine 112) can model performance as a response to load. From module 418, flowchart 400 returns to module 406 and continues as previously described.

[0059] It can be noted that modules 404, 416, and 418 can be processed in parallel; however, for practical purposes, if the model or prediction is not updated relative to the model or prediction from the first loop, one or more modules can be skipped in the second loop from module 414 to modules 404, 416, and 418. Of course, modules 404, 416, and 418 can also be rearranged for serial processing. It can also be noted that if deployment is performed without SLA metric data (which will be provided later in the process), modules 404, 416, and 418 may conceivably precede module 402. Finally, it can be noted that module 402 can be repeated if the declarative performance purpose changes (not shown).

[0060] Figure 5A flowchart 500 depicts an example of generating predictive autoscaling and resource optimization results in conjunction with a machine learning process. Flowchart 500 begins with a module 502 that monitors performance indicators and resource usage. A human or artificial agent of the service provider (or service consumer) can, for example, provide new performance indicators after reviewing the performance of converged deployments. A declarative performance interface engine (e.g., combined with...) Figure 1 The described declarative performance interface engine (108) can monitor performance indicators. It also includes a convergence deployment and resource metric collection engine (e.g., combined with...). Figure 1 The described convergence deployment, resource, and application-level metrics collection engine (120) can monitor resource usage.

[0061] Flowchart 500 continues to module 504, which predicts application load and seasonality. Seasonality can be illustrated with a use case, in this example, a footwear e-commerce deployment. Successful footwear e-commerce deployments typically have stable traffic with some seasonality during Black Friday and holiday seasons, plus some seemingly random spikes (e.g., when new shoes are released). Reactive autoscaling is suboptimal during these times due to the importance of performance to revenue. Systems engineers would manually overproduce to meet SLOs (Supply Limits). As discussed in this paper, predictive autoscaling and resource optimization systems can learn these seasonalities and provide the correct amount of resources (with supply insurance) for these events without human intervention. Furthermore, during the year, when new shoes are released, systems engineers are often unprepared for the large loads during the release period. When they reactively scale vertically to meet demand, the shoes sell for 10 times the price on eBay. As discussed in this paper, predictive autoscaling and resource optimization systems can predict these seemingly random traffic spikes and scale accordingly to meet SLOs and achieve maximum revenue and customer satisfaction. Advantageously, money is not wasted on overproducing simply to prepare for these events. Application load prediction engine (e.g., combined with) Figure 1 The described application load prediction engine (114) can predict application load and seasonality.

[0062] Flowchart 500 continues to module 506, which focuses on learning the application's behavior under load. This includes a dynamic estimation engine (e.g., combined with...). Figure 1 The described dynamic estimation engine (112) can learn the behavior of an application under load.

[0063] Flowchart 500 continues to module 508, which estimates the resources used under the predicted demand for resources. This includes the application load prediction engine (e.g., combined with...). Figure 1 The described application load prediction engine 114 can estimate the resources used under predicted demand for resources.

[0064] Flowchart 500 continues to module 510, which estimates the prediction pattern used to set resource limits. Application load prediction engine (e.g., combined with...) Figure 1 The described application load prediction engine 114 can estimate prediction patterns used to set resource limits.

[0065] Flowchart 500 continues to module 512, which minimizes the resources required to meet the forecasted demand. This includes a minimum cost optimization engine (e.g., combined with...). Figure 1 The described minimum cost optimization engine (116) can minimize the resources required to meet forecasted needs.

[0066] Flowchart 500 ends at module 514, which learns from the decisions made to improve predictions and resource estimates. This extends the resource optimization executor engine (e.g., by combining...). Figure 1 The described extended resource optimal configuration executor engine (118) can benefit from learning from decisions made during convergent deployment to improve forecasting and resource estimation.

[0067] Figure 6 Figure 600 illustrates an example of a system used to generate minimum cost optimization parameters. Figure 600 includes: an SLA metric data storage area 604, which can be implemented to combine... Figure 1 The SLA metrics data storage area 104 is described; the declarative performance interface engine 608, which is coupled to the SLA metrics data storage area 604 and can be implemented as a combination Figure 1 The declarative performance interface engine 108 is described; the dynamics estimation engine 612, which can be implemented as a combination of Figure 1 The described dynamic estimation engine 112; application load prediction engine 614, which can be implemented as a combination Figure 1 The described application load prediction engine 114; and the lowest cost optimization engine 616, which can be implemented as a combination of Figure 1The description includes a minimum cost optimization engine 116; a declarative performance data storage area 626 coupled to a declarative performance interface engine 608 and a minimum cost optimization engine 616; a behavioral model data storage area 628 coupled to a dynamic estimation engine 612 and a minimum cost optimization engine 616; a performance model data storage area 630 coupled to a dynamic estimation engine 612 and a minimum cost optimization engine 616; a convergence deployment and resource metric data storage area 632 coupled to a dynamic estimation engine 612 and an application load prediction engine 614; a pattern learning engine 634 coupled to the convergence deployment and resource metric data storage area 632; a prediction model data storage area 636 coupled to the application load prediction engine 614 and the pattern learning engine 634; a prediction load data storage area 638 coupled to the application load prediction engine 614 and the minimum cost optimization engine 616; and a minimum cost optimization parameter data storage area 640 coupled to the minimum cost optimization engine 616.

[0068] The declarative performance interface engine 608 converts SLA metrics from the SLA metric data store 604 into a declarative performance data structure represented by the declarative performance data store 626. The declarative performance interface engine 608 may or may not accept instructions from a human or artificial agent from the service provider (or service consumer) to populate the SLA metric data store 604. If the SLA metric data store 604 is modified, the declarative performance engine 608 transforms the modification to match the expected SLO represented in the SLA metric data store 604 by the declarative performance data structure in the declarative performance data store 626.

[0069] The dynamic estimation engine 608 uses machine learning techniques such as deep learning to generate a behavioral model represented by the behavioral model data storage area 628 and a performance model represented by the performance model data storage area 630. The model can be improved through feedback associated with applicable convergence deployments. This feedback is represented by the convergence deployment and resource metric data storage area 632. The convergence deployment and resource metric data storage area 632 can be populated by a convergence deployment and resource metric collection engine (not shown), which can be implemented to combine... Figure 1 The description refers to the convergence of deployment, resource, and application-level metrics collection engines 120.

[0070] The pattern learning engine 634 uses deep learning to understand workloads to generate models for seasonal, popular, bursty, and random workloads. Based on requests and the understanding of workloads, signature limits can be set on deployments according to the utilization patterns (e.g., bursty workloads result in higher limits, while stable workloads result in lower limits). The result of deep learning is a predictive model represented by the predictive model data store 636. The predictive model data store 636 can be improved using feedback associated with applicable convergent deployments. This feedback is represented by the convergent deployment and resource metrics data store 632.

[0071] The application load prediction engine 614 uses one or more prediction models from the prediction model data store 636 and feedback from the convergence deployment and resource metrics data store 632 to estimate resource usage for future times; the predicted load is represented by the predicted load data store 638.

[0072] The minimum cost optimization engine 616 uses a declarative performance data storage area 626, a behavioral model data storage area 628, a performance model data storage area 630, and a predicted load data storage area 638 to generate minimum cost optimization parameters represented by the minimum cost optimization parameter data storage area 640. The minimum cost optimization parameters can be used by a software deployment platform, which may include, for example, an extended resource optimal configuration executor engine (not shown), which can be implemented to combine... Figure 1 The described extended resource optimal configuration executor engine 118.

Claims

1. A predictive autoscaling and resource optimization system comprising: a declarative performance interface engine configured to: receive service level agreement (SLA) indicators; convert the SLA indicators into a declarative performance data structure representing declared performance goals for a software deployment; and monitor performance indicators of the deployed software; a converged deployment, resource, and application level metrics collection engine configured to monitor resource usage of the deployed software; a dynamic estimation engine configured to generate an application behavior model based on performance indicators and resource usage as a function of load; an application load prediction engine configured to predict load at a future time; a least cost optimization engine configured to generate least cost optimization parameters based on the declarative performance data structure, the application behavior model, and predicted load; a predictive autoscaling and resource optimization operator engine configured to provide the least cost optimization parameters to a scaling resource optimal configuration enforcer engine; a scaling resource optimal configuration enforcer engine configured to enforce a converged deployment; a load distribution and metrics engine configured to perform load balancing on one or more of traffic to and traffic from the converged deployment, the load distribution and metrics engine configured by the scaling resource optimal configuration enforcer engine according to the least cost optimization parameters; the converged deployment, resource, and application level metrics collection engine further configured to: monitor resources associated with the converged deployment; and provide feedback associated with the converged deployment to the dynamic estimation engine; and the dynamic estimation engine further configured to generate an updated application behavior model based on the feedback.

2. The system of claim 1, wherein, the SLA indicators comprise one or more of service level indicator (SLI) indicators and service level objective (SLO) indicators.

3. The system of claim 1, wherein, the SLA indicators are defined by a human agent.

4. The system of claim 1, wherein, the SLA indicators are defined by an artificial agent.

5. The system of claim 1, wherein, the performance indicators comprise one or more of request count and request duration.

6. The system of claim 1, wherein, the resource usage comprises usage of one or more of memory, CPU power, disk I / O, and network I / O.

7. The system of claim 1, wherein, the application load prediction engine is configured to predict one or more of seasonal load, popular load, burst load, and random load.

8. The system of claim 1, wherein, the application load prediction engine is configured to estimate a prediction pattern for setting resource limits.

9. The system of claim 1, wherein, the dynamic estimation engine generates the application behavior model using deep learning.

10. The system of claim 1, further comprising a predictive autoscaling and resource optimization operator engine configured to determine a measurement error according to the declared performance goals.

11. A predictive autoscaling and resource optimization method comprising: receiving service level agreement (SLA) indicators; converting the SLA indicators into a declarative performance data structure representing declared performance goals for a software deployment; monitoring performance indicators and resource usage of the deployed software; generating an application behavior model based on performance indicators and resource usage as a function of load; predicting load at a future time; generating a least cost optimization parameter based on the declarative performance data structure, the application behavior model, and the predicted load; performing a converged deployment; performing load balancing on one or more of traffic to and from the converged deployment according to the least cost optimization parameter; monitoring resources associated with the converged deployment; providing feedback associated with the converged deployment; and generating an updated application behavior model based on the feedback.

12. The method of claim 11, wherein, The SLA metrics include one or more of service level indicator (SLI) metrics and service level objective (SLO) metrics.

13. The method of claim 11, wherein, The SLA metrics are defined by a human agent.

14. The method of claim 11, wherein, The performance indicators include one or more of request count and request duration.

15. The method of claim 11, wherein, The resource usage includes usage of one or more of memory, CPU power, disk I / O, and network I / O.

16. The method of claim 11, wherein, The predicted load includes one or more of seasonal load, popular load, burst load, and random load.

17. The method of claim 11, further comprising estimating a predicted pattern for setting resource limits.

18. The method of claim 11, wherein, The application behavior model is generated using deep learning.

19. The method of claim 11, further comprising determining a measurement error according to the declared performance objectives.

20. A predictive autoscaling and resource optimization system, comprising: means for receiving service level agreement (SLA) metrics; means for converting the SLA metrics into a declarative performance data structure representing declared performance objectives of a software deployment; means for monitoring performance indicators and resource usage of a deployed software; means for generating an application behavior model based on the load-dependent performance indicators and resource usage; means for predicting load at a future time; means for generating a least cost optimization parameter based on the declarative performance data structure, the application behavior model, and the predicted load; means for performing a converged deployment; means for performing load balancing on one or more of traffic to and from the converged deployment according to the least cost optimization parameter; means for monitoring resources associated with the converged deployment; means for providing feedback associated with the converged deployment; and means for generating an updated application behavior model based on the feedback. ​

Images