DETECTING AND HANDLING EXCESSIVE RESOURCE UTILIZATION IN A DISTRIBUTED COMPUTING ENVIRONMENT - Patent application

Abstract

A method, computer program product, and computer system are provided for predicting excessive resource utilization in a distributed computing environment. A processor obtains at least a portion of code associated with a workload in the distributed computing environment. The processor obtains accounting information associated with the workload. The processor determines a likelihood that the workload exhibits excessive resource utilization based at least in part on the at least a portion of the code and the accounting information associated with the workload. In response to the likelihood of excessive resource utilization exceeding a threshold, the processor reschedules the workload in the distributed computing environment.

Description

[Technical field]

[0001] The present invention relates generally to the field of resource provisioning, and more particularly to limiting resource usage to workloads in a distributed computing environment. [Background technology]

[0002] A distributed computing environment provides convenient, on-demand access to a shared pool of configurable computing resources that can be rapidly provided to users of the platform. The amount and type of resources are likely to vary widely between users based on the computing needs they require. Resource provisioning is the act of a distributed computing environment provider detecting and satisfying the needs of each user to provide seamless access to the vast computing power provided by the distributed computing environment. Summary of the Invention [Means for solving the problem]

[0003] Embodiments of the present invention provide a method, computer system, and computer program product for predicting excessive resource usage in a distributed computing environment. A processor obtains at least a portion of code associated with a workload in the distributed computing environment. The processor obtains accounting information associated with the workload. The processor determines a likelihood that the workload exhibits excessive resource usage based at least in part on the at least a portion of the code and the accounting information associated with the workload. In response to the likelihood of excessive resource usage exceeding a threshold, the processor reschedules the workload in the distributed computing environment. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 illustrates a distributed computing environment according to one embodiment of the invention. [Diagram 2] FIG. 2 illustrates abstraction model layers according to one embodiment of the present invention. [Diagram 3] FIG. 3 illustrates a distributed computing environment for scheduling workloads and detecting excessive resource utilization according to one embodiment of the present invention. [Figure 4] FIG. 4 illustrates an operational process of an inspector and scheduler for identifying excessive resource utilization and scheduling workloads in a distributed computing environment according to one embodiment of the present invention. [Diagram 5] FIG. 5 illustrates an operational process of an inspector that determines the likelihood that a workload will be resource overloaded, according to one embodiment of the present invention. [Figure 6] FIG. 6 illustrates a block diagram of components of a computing device executing an inspector and a scheduler according to one exemplary embodiment of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Distributed computing platforms provide users with unimaginable amounts of computing resources, but excessive use of these resources is likely to impact other users as well as the overall efficiency of the platform. By detecting and responding to users who deploy excessive amounts of resources, the efficiency of the platform can be significantly improved. Embodiments of the present invention improve upon prior solutions of excessive resource usage in distributed computing platforms by looking not only at users' activities, but also by incorporating more proactive measures and inspection of the code and resource usage that users use from the platform.

[0006] The present invention may be a system, method, or computer program product, or any combination thereof. The computer program product may include one or more computer readable storage media having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0007] The computer readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0008] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to an individual computing device / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer readable program instructions from the network and transmits the computer readable program instructions to the individual computing device / processing device for storage in a computer readable storage medium.

[0009] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, such as Smalltalk, C++, and the like, conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0010] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer readable program instructions.

[0011] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to produce a machine. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer programmable data processing apparatus or other device, or combinations thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof.

[0012] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0013] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, depending on the functions involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, as well as combinations of multiple blocks of the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special purpose hardware and computer instructions.

[0014] Although the present disclosure includes detailed descriptions of cloud computing, it should be understood in advance that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0015] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0016] Its features are as follows.

[0017] On-demand self-service: A cloud consumer can unilaterally provision computing capacity, such as server time and network storage, as needed, without the need for human interaction with the provider of the service.

[0018] Broad Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0019] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the provided resources, but are said to be location-independent in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0020] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, quickly scaled out, quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning are often unlimited and can be purchased in any quantity at any time.

[0021] Measured Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0022] The service models are as follows:

[0023] Software as a Service (SaaS): The capability offered to consumers to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, e.g., a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0024] Platform as a Service (PaaS): A capability offered to a consumer to deploy consumer-created or acquired applications, created using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, e.g., including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application-hosting environment configuration.

[0025] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selecting network components (e.g., host firewalls).

[0026] The deployment models are as follows:

[0027] Private Cloud: Cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party, and may be on-premises or off-premises.

[0028] Community Cloud: Cloud infrastructure is shared by several organizations and supports a particular community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party, and may reside on-premise or off-premise.

[0029] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0030] Hybrid Cloud: The cloud infrastructure remains a unique entity, but is a hybrid of two or more clouds (private, community, or public) brought together by standardized or specialized technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0031] A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0032] Referring now to FIG. 1, an exemplary cloud computing environment 50 is illustrated. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described herein (not shown). This allows the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It is understood that the types of computing devices 54A-54N illustrated are intended to be exemplary only, and that the computing nodes 10 and the cloud computing environment 50 communicate with any type of computerized device (e.g., using a web browser) over any type of network or network addressable connection or combinations thereof.

[0033] Referring now to Figure 2, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 1) is shown. It should be understood that the components, layers and functions shown in Figure 2 are intended to be merely exemplary, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0034] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (Reduced Instruction Set Computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0035] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71; virtual storage 72; virtual networks 73, including, for example, virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0036] In one example, management layer 80 may provide multiple functions described below. Metering and pricing 81 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks and protection for data and other resources. User portal 82 provides access to the cloud computing environment for consumers and system administrators. Service level management 83 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 84 provides pre-allocation and procurement for cloud computing resources where future requirements are predicted according to SLAs. Resource provisioning 85 provides dynamic procurement of computing resources and other resources utilized to execute tasks within the cloud computing environment.

[0037] In various embodiments and scenarios, resource provisioning 85 provides workload inspection and workload scheduling. As discussed herein, workload inspection provides monitoring of various resources and activities associated with various workloads represented by workload tier 90. Workload scheduling provides scheduling and resource access for various workloads as represented by workload tier 90.

[0038] Workload tier 90 provides examples of functions for which the cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91; software development and lifecycle management 92; virtual classroom instructional delivery 93; data analytics processing 94; transaction processing 95; and mobile desktop 96.

[0039] 3 illustrates a distributed computing environment 100 for scheduling workloads 120a-120n. The distributed computing environment 100 is a multi-tenant computing environment in which multiple users with accounts 124a-124n provide code 122a-122n to be executed by the workload layer 90 as workloads 120a-120n. The workloads 120a-120n are portable and independent collections of code that can be executed by various hardware and software components of the hardware and software layer 60. The workloads 120a-120n may be executed by various devices, platforms, or services provided by the hardware and software layer 60, making the workloads 120a-120n portable. In addition, the workloads 120a-120n include code, libraries, and other files necessary to execute the workloads on various devices and platforms, making the workloads 120a-120n independent. The code 122a-122n for each workload 120a-120n includes the code or any machine-readable script, batch, or set of instructions, as well as any user-provided support libraries associated with the workload.

[0040] In various embodiments, the workloads 120a-120n are stored as containers or logical packages that allow them to be portable and independent of the device or environment in which they run. In some embodiments, the workloads 120a-120n are implemented in a Kubernetes container orchestration system that packages the code 122a-122n that can run on various platforms and services provided by the hardware and software layer 60. Those skilled in the art will appreciate that any containerization method may be used to provide portability of workloads in the distributed computing environment 100 without departing from this invention.

[0041] In various embodiments, the distributed computing environment 100 provides functionality and mechanisms for users to create accounts. Each workload 120a-120n has an associated account that includes various account information, such as, but not limited to, account history, account age, account type, frequency of resource deployment, frequency of resource removal, prior quarantines, or other accounts with similar information to the account information associated with the workload. Account history indicates the types of applications, code, and services that the account has accessed or requested in the distributed computing environment 100. Account age indicates the time the account was created and the period of time the account has been active. Account type indicates the type of account that the user created or selected (e.g., paid, free, or any other type of tiering for services provided by the distributed computing environment 100). Resource deployment frequency indicates the percentage in the account history that the user account has made requests to deploy resources or run workloads in the distributed computing environment 100. The frequency of resource removal is the percentage in the account history that the user account made requests to remove resources from a workload or the percentage that the user account terminated a workload. Prior quarantine also provides the current quarantine status of the account, as discussed herein, as well as any previous changes in the quarantine status of the account. In addition, the account information may include information indicative of other accounts with similar information to the user account (e.g., accounts created at approximately the same time, e.g., within an hour, as well as accounts created from IP addresses associated with a similar geographic location).

[0042] In various embodiments, the scheduler 110 is a service of the management layer 80 that facilitates and assigns execution to the virtualization layer 70 of various ongoing workloads 120a-120n currently being processed by the distributed computing environment 100. The scheduler 110 identifies various virtualization hosts within the virtualization layer 70 based on various factors, such as the capabilities and availability of the hosts. In addition, as discussed herein, the scheduler 110 includes a quarantine module 112 that alters or alternatively otherwise alters the scheduling of the workloads 120a-120n based on a determined likelihood of excessive resource usage associated with the workloads 120a-120n, as well as quarantines or stops the workloads 120a-120n in some scenarios.

[0043] In some embodiments, each workload 120a-120n includes a respective inspector 126a-126n that inspects the respective code 122a-122n for the respective workload (e.g., the inspectors 126a-126n are services embedded in containers and executed by hosts in the virtualization layer 70). In other embodiments, the inspector 126 is a service of the management layer 80 that inspects the respective code 122a-122n for each workload 120a-120n. In various embodiments, the inspector 126 compares the respective code 122a-122n for each workload 120a-120n to the code repository 132 to determine whether the respective code 122a-122n contains any known patterns of excessive resource usage. The inspector 126 reviews the code 122 of each of the workloads 120a-120n for various patterns that are provided in the code repository 132 as indicative of potential excessive resource usage.

[0044] One aspect of the Inspector 126 review is string patterns present in the code 122. For example, function calls, uniform resource locators (URLs), and other components of the code associated with excessive resource usage are compared by the Inspector 126 to a code repository 132, where the code repository 132 contains known string patterns that indicate potential excessive resource usage. Another aspect that the Inspector 126 reviews is process patterns. Some code 122 may contain various threads or processes that contribute to or service other parts of the code 122, where the code repository 132 contains names of known processes or threads that indicate potential excessive resource usage. Another aspect of the Inspector 126 review is file patterns. Some code 122 may generate output files that may provide the final output to a user of the executed workload, or may be passed to another component of the code 122. Aspects such as filename, size, and content are compared by inspector 126 to a code repository 132, which contains known output or intermediate files that exhibit potential excessive resource usage.

[0045] Another aspect of the inspector 126's review is the operational activity of the workload 120 of the workloads 120a-120n. When a workload is deployed in the distributed computing environment 100, various activities occur that consume or use various resources of the hardware and software layer 60. For example, when a workload 120 is deployed by the management layer 80 and hosted by the virtualization layer 70, a certain percentage of one or more central processing units (CPUs) of the devices in the hardware and software layer 60 are utilized. As another example, some workloads may require upload and download bandwidth from networking components in the hardware and software layer 60. In either example, the inspector 126 monitors such operational activity of the workload 120.

[0046] In various embodiments, the inspector 126 evaluates each account 124 associated with each workload 120 of the workloads 120a-120n based on various account information discussed herein, such as, but not limited to, account data 134 including account history, account age, account type, frequency of resource deployment, frequency of resource removal, previous quarantines, or other accounts with similar information to the account information associated with the workload. As discussed herein, some activities of excessive resource usage in multi-tenant computing utilize multiple accounts to be effective. By reviewing and determining excessive resource usage based on account information, the inspector 126 can detect and address more potential excessive resource usage from the workloads 120a-120n.

[0047] In various embodiments, the inspector 126 determines the likelihood that a workload may be associated with excessive resource usage. The inspector 126 separates various information from string patterns, process and file patterns, operational activity and account information associated with the workload 120 into vectors, where each element or dimension of the vector is associated with a selected item of the above information. For each item, the inspector 126 determines the likelihood that the item of information indicates resource overage. In the case of string patterns, process patterns and file patterns, the inspector 126 compares the code 122 of the workload 120 to the code repository 132 to identify and match patterns indicative of potential excessive resource usage associated with the code 122. In the case of operational activity, the inspector 126 monitors the network and resource usage of the workload 120 for exceeding certain thresholds or matching usage patterns indicative of excessive resource usage associated with the code 122.

[0048] In the case of account information, inspector 126 identifies patterns in the particular account associated with the workload, as well as other accounts stored in account data 134. In various embodiments, inspector 126 compares account 124 to other accounts in account data 134 to determine whether account 124 shares any patterns associated with previously identified over-resourced accounts in account data 134. For example, inspector 126 compares the number and type of applications, services, or other features of workloads 120a-120n currently deployed or that have been deployed by account 124 to other over-resourced accounts in account data 134. For accounts 124a-124n that match a similar deployment pattern as an over-resourced account in account data 134, inspector 126 determines that potential over-resource usage is occurring with respect to this dimension of the likelihood vector.

[0049] In various embodiments, inspector 126 compares account creation information to other accounts in account data 134 to identify similar accounts. For example, inspector 126 compares account 124 to account data 134 to identify the number of accounts created within a certain time frame (e.g., some users may create multiple accounts in succession to orchestrate multiple workloads that provide malicious attacks, e.g., denial of service attacks, that would require a large amount of computing resources of a distrusted computing environment 100). In addition, inspector 126 compares login or account creation location, IP address, or other geolocation data associated with account 124 to account data 134 to determine whether account 124 shares a similar access location from the user (e.g., some users may create multiple accounts that originate from the same device, the same network, or the same location).

[0050] In some scenarios, some users targeting a multi-tenant distributed computing environment for computing resources create various accounts to coordinate large-scale actions within the untrusted computing environment or to maintain access to the system if one account is compromised. In various embodiments, the inspector 126 compares various patterns of use of the accounts and access to the untrusted computing environment 100 to determine whether the accounts have similar usage patterns that may indicate they are owned or operated by a single user or group of users. The inspector 126 compares usage patterns, such as similar login locations (e.g., accounts accessed within a given IP address range or similar geographic location), time of login and access to the accounts (e.g., a coordinated group of accounts operated by a malicious actor may be accessed sequentially or all at once where the actor performs actions and uploads code for each account), or duration of login or other activity (e.g., a malicious actor may rapidly upload code to various accounts and then log out after the upload is complete).

[0051] In various embodiments, the inspector 126 selects one or more dimensions for a likelihood vector that indicates across the string, process and file patterns, operational activities, and account information associated with the workload 120 if the account and associated workload may perform activities that would result in excessive resource usage within the distributed computing environment 100. For example, the inspector 126 generates a likelihood vector with the following dimensions: [Code_Patterns, File_Patterns, Process_Patterns, Network_Patterns, Account_Consistency, Account_Similarity, Account_Type, Account_Creation]. The Code_Patterns, File_Patterns, Process_Patterns, and Network_Patterns reflect the likelihood that the code, file, process, and network activities associated with the workload may indicate excessive resource usage. Account_consistency, Account_similarity, Account_type, and Account_creation reflect the likelihood that various account information associated with a workload may indicate excessive resource utilization. In various scenarios, each element of the vector indicates the likelihood that each element indicates possible excessive resource usage. For example, the resulting likelihood vector may have the following values: [0 (code_pattern), 0.2 (file_pattern), 0.3 (process_pattern), 0.1 (network_pattern), 0.2 (account_consistency), 0.9 (account_similarity), 0.8 (account_type), 0.8 (account_creation)]. In this above example, inspector 126 determines that there is a likelihood score of 0.9 that this workload has an account similar to other known resource-excessive accounts, indicating that the account may use excessive resources based on this dimension.Conversely, inspector 126 may determine that there is a likelihood score of 0.1 for a workload's network pattern that matches resource-excessive network activity, indicating that the network activity of that workload is likely to over-utilize resources.

[0052] Those skilled in the art will appreciate that the likelihood vectors generated by inspector 126 may include or exclude any dimensions without departing from the invention, as discussed herein. In addition, in some embodiments, inspector 126 is equipped with a machine learning perspective that automatically selects and removes dimensions in the likelihood vectors based on training and prior determinations of likely over-utilization of resources by both workloads and accounts, as discussed herein.

[0053] In some embodiments, inspector 126 determines values ​​for the likelihood vector based on analysis of particle filters or sequential Monte-Carlo methods. Monte Carlo particle filters use a set of particles, or samples, to represent the distribution of a stochastic process or process that leads to which actions in this scenario may indicate excessive use of resources. Particle filter techniques provide a process for creating samples from the required distributions without requiring assumptions about state space models or state distributions.

[0054] For example, inspector 126 selects account_consistency as an element in the likelihood vector. Inspector 126 builds a vector for the element of account_consistency, expressed as [(number of applications), (average memory usage), (number of services)], which is built for each account to create a series of vector samples. Taken together, the vector samples form a Gaussian distribution. When a new account creates or updates a new application in distributed computing environment 100, inspector 126 calculates the account_consistency likelihood of the new application by comparing the vector for the new application to the Gaussian distribution generated from combining the past account_consistency vectors of all other known applications. Based on this comparison, inspector 126 determines the likelihood that the new application falls within the entire distribution, generating a value between 0 and 1.

[0055] Those skilled in the art will appreciate that the historical Gaussian distribution for account_consistency may change as new accounts are created and new applications are updated. In various embodiments and scenarios, the inspector 126 includes both supervised and unsupervised machine learning and training to adjust and account for such variations. For example, the inspector 126 may have a certain threshold for excessive resource usage, such as an upper limit for memory usage. The inspector 126 determines a threshold that corresponds to a certain percentage of the historical Gaussian distribution (e.g., 95% of the applications use memory) based on historical usage of memory for various workloads. In various embodiments and scenarios, the inspector 126 modifies the threshold based on changes in usage over time.

[0056] In other scenarios and embodiments, inspector 126 includes machine learning to select and remove dimensions from the likelihood vector. For example, one or more historical workloads have been analyzed and found to use excessive amounts of resources in distributed computing environment 100. Inspector 126 compares the workload to a current model based on the classification of the historical workload as being resource-excessive. If the current model fails to properly classify the historical workload as resource-excessive, inspector 126 selects one or more dimensions to remove or add to the likelihood vector. If the updated vector has a higher score, inspector 126 includes the dimension in future calculations. Similarly, if removing a dimension increases the overall score, inspector 126 excludes the dimension in future calculations.

[0057] In various embodiments, the inspector 126 combines each of the likelihood dimension values ​​to create an overall excess resource usage likelihood score for the workload 120 in the distributed computing environment 100. The inspector 126 applies a weighted average to the dimensions of the likelihood vector. The inspector 126 compares the weighted average to a threshold. For values ​​above the threshold, the inspector 126 classifies the workload 120 as potentially over-resourced and sends instructions to the scheduler 110, which places the workload 120 in tiered stages of quarantine. For example, the scheduler 110 has four tiers of quarantine. For workloads with excess resource usage likelihood scores less than 0.2, the scheduler 110 places no limits or restrictions on the workload 120. For workloads with an excessive resource usage likelihood score below 0.45, the scheduler 110 places some limitations on resource usage but keeps the workload 120 in a trusted cluster of the distributed computing environment 100. For workloads with a score higher than the previous tier (e.g., above 0.45), the scheduler 110 places the workload 120 in a special quarantine cluster that may limit the services and other features provided by the distributed computing environment 100. And for workloads exceeding a fourth threshold (e.g., 0.9), the scheduler 110 terminates the workload. Additionally, for workloads with a large excessive resource usage likelihood score (e.g., above 0.90), the inspector 126 includes the workload 120 and the account 124 data in future machine learning and classification training sessions. Those skilled in the art will appreciate that various threshold tiers and scheduling of the workloads 120a-120n may be used without departing from the present invention.

[0058] In some embodiments, scheduler 110 provides isolation hierarchies categorized based on the types of excess resources that it determines a workload is likely to be overutilized for. For example, one isolation hierarchy that scheduler 110 maintains is a cluster for workloads likely to use excess CPU resources, while another hierarchy is for workloads likely to use excess network resources. In some embodiments and scenarios, scheduler 110 isolates likely resource-overutilizing workloads into a tier associated with the dimension of the likelihood vector that has the largest value.

[0059] In another embodiment, scheduler 110 creates isolation hierarchies of workloads with high likelihood of resource overload to stagger or mitigate the impact of resource overload that the workloads may incur. Scheduler 110 groups resource overload workloads such that the overall average of the resource dimensions is balanced, rather than the workloads' likelihood dimensions being concentrated in a single dimension. For example, scheduler 110 might place a memory resource overload workload and a CPU resource overload workload in the same tier, while placing another memory resource overload workload in a different tier, to balance the impact of the clustered workloads on the tiers.

[0060] In various embodiments, inspector 126 continues to evaluate workloads 120a-120n for likelihood of resource over-utilization. If a given workload changes operational perspective (e.g., new code 122) or a new excessive resource utilization likelihood score is otherwise determined (e.g., such updated training of the machine learning perspective of inspector 126), inspector 126 assigns a new excessive resource utilization likelihood score to the workload and scheduler 110 reschedules the workload accordingly.

[0061] 4 illustrates an operational process, generally designated 200, of inspector 126 and scheduler 110, which identify excessive resource usage and schedule workloads in distributed computing environment 100. In steps 202 and 204, inspector 126 retrieves code and accounting information associated with workloads 120a-120n that are scheduled to execute or are executing in distributed computing environment 100. In some scenarios, inspector 126 is included within a containerized workload 120, which includes code and libraries for inspecting workload 120, underlying code 122, and associated accounts 124. In other embodiments, inspector 126 is a service of distributed computing environment 100 that operates in conjunction with other services of distributed computing environment 100, such as scheduler 110.

[0062] In step 202, inspector 126 retrieves code 122 for workload 120, workloads 120a-120n. Code 122 may include, but is not limited to, any program code syntax, libraries, or other support software required to execute workload 120. In process 204, inspector 126 retrieves account information associated with account 124 to which code 122 is being uploaded, submitted, or otherwise provided for execution in distributed computing environment 100. In process 206, inspector 126 determines the likelihood of excessive resource usage for workload 120, which is described in detail in FIG.

[0063] 5 illustrates an operational process, generally designated 300, of an inspector 126 for determining the likelihood that a workload is over-resourced. In process 302, the inspector 126 selects from various factors associated with the code 122 and the account 124 for inspection. As discussed herein, the inspector 126 evaluates the code 122 based on one or more of, but not limited to, string patterns, process patterns, file patterns, network patterns, or resource patterns associated with the syntax of the code 122, and performance and other usage patterns associated with the execution of the code 122 in the distributed computing environment 100. The inspector 126 evaluates the account 124 based on, but not limited to, one or more of account history, account age, account type, frequency of resource deployments, frequency of resource removals, previous quarantines, or other accounts having similar information to the account information associated with the workload.

[0064] In process 302, inspector 126 selects from the above dimensions related to code 122 and accounts 124 associated with workload 120. In some embodiments, inspector 126 includes all of the above or a predetermined selection. In other embodiments, inspector 126 changes the selected dimensions over time and based on changing historical data. For example, inspector 126 includes a machine learning model that is updated based on changing historical data of previous workloads that trains inspector 126 to choose different dimensions as classifiers for excessive resource usage.

[0065] In process 304, the inspector 126 determines the historical distribution of previous workloads to determine whether the currently evaluated workload 120 matches previous excess resource usage in the historical distribution of previous workloads. For example, the inspector 126 may include one or more particle filters and perform a sequential Monte-Carlo analysis of previous workloads to determine the historical distribution of values ​​for the selected dimension. In process 306, the inspector 126 determines a resource excess likelihood for each dimension. The inspector 126 determines the likelihood that the workload will be resource excess based on the historical distribution of previous workloads determined in process 304. For example, in a scenario where a particle filter is utilized in process 304, the inspector 126 generates a Gaussian distribution or a bell-shaped curve with a given mean and variance. The inspector 126 compares the value associated with the workload 120 for the dimension being evaluated to the Gaussian distribution determined in process 304. For workloads with values ​​close to (e.g., within a predetermined distance of) the mean or center point of the Gaussian distribution, inspector 126 determines a low resource excess likelihood value for the selected dimension. For workloads with values ​​further away (e.g., at least a predetermined distance from) the mean, inspector 126 determines a high resource excess likelihood value for the selected dimension.

[0066] In process 308, Inspector 126 determines a resource over-likelihood for the workload being evaluated. In some scenarios, Inspector 126 calculates a weighted sum of each dimension selected in process 302, where the weights are trained and modified based on historically trained machine learning models. In other scenarios, Inspector 126 evaluates the value of each dimension, and if the dimension is above a predefined threshold, Inspector 126 assigns a high resource over-likelihood for the workload, regardless of the values ​​of the other dimensions. For example, if the Code_Pattern dimension is determined to be above 95%, Inspector 126 determines a resource over-likelihood of 100% for workload 120.

[0067] Referring back to FIG. 4, in process 208, inspector 126 compares the determined resource excess likelihood for a workload to one or more predefined thresholds. For example, inspector 126 has various tiers for various escalations of quarantine for the workload. If the resource excess likelihood for the workload exceeds any threshold (the “Yes” branch of process 208), inspector 126 sends the resource excess likelihood to scheduler 110 to reschedule the workload (process 212). Scheduling includes one or more of the following, but is not limited to: placing the workload in a trusted or quarantined zone of distributed computing environment 100, restricting or limiting access to resources or services, or terminating the workload. If the resource excess likelihood has not changed by more than one threshold (the "No" branch of process 208), i.e., if the resource excess likelihood is less than or equal to a predefined threshold, inspector 126 instructs scheduler 110 to maintain the current scheduling for the workload (process 210). In some embodiments, inspector 126 passes the resource excess likelihood for the workload to scheduler 110, thereby modifying the scheduling of the workload, as discussed herein.

[0068] 6 illustrates a block diagram 600 of components of scheduler 110 and inspectors 126a-126n in accordance with an exemplary embodiment of the invention. It should be understood that FIG. 6 is intended to provide only an example of one implementation and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0069] Scheduler 110 and inspectors 126a-126n include a communications fabric 602 that provides communication between one or more computer processors 604, memory 606, persistent storage 608, a communications unit 610, and one or more input / output (I / O) interfaces 612. Communications fabric 602 can be implemented with any architecture designed to pass data or control information, or a combination thereof, between processors (e.g., microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 602 can be implemented using one or more buses.

[0070] Memory 606 and persistent storage 608 are computer-readable storage media. In this embodiment, memory 606 includes random access memory (RAM) 614 and cache memory 616. In general, memory 606 may include any suitable volatile or non-volatile computer-readable storage medium.

[0071] The scheduler 110 and the inspectors 126a-126n include data and programs for performing the methods discussed herein that are stored in persistent storage 608 for execution and / or access by one or more of the respective computer processors 604 via one or more memories of memory 606. In this embodiment, the persistent storage 608 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, the persistent storage 608 can include a solid state hard drive, a semiconductor memory device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0072] The media used by persistent storage 608 may also be removable. For example, a removable hard disk may be used for persistent storage 608. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 608.

[0073] In these examples, the communications unit 610 provides for communication with other data processing systems or devices, such as those described above that comprise the resources of the distributed computing environment 100. In these examples, the communications unit 610 includes one or more network interface cards. The communications unit 610 may provide communications through the use of either or both physical and wireless communication links. Various data and programs may be downloaded to the persistent storage 608 via the communications unit 610 to perform the methods discussed herein.

[0074] One or more I / O interfaces allow for input and output of data with other devices that may be connected to the scheduler 110 and the inspectors 126a-126n. For example, the I / O interface 612 may provide a connection to an external device 618, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. The external device 618 may also include a portable computer-readable storage medium, such as a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to practice embodiments of the present invention may be stored on such a portable computer-readable storage medium and may be loaded onto the persistent storage 608 via the one or more I / O interfaces 612. The one or more I / O interfaces 612 also connect to a display 620.

[0075] Display 620 provides a mechanism for displaying data to a user and may be, for example, a computer monitor or television screen.

[0076] The programs described herein are identified based on the applications for which they are implemented in particular embodiments of the invention. It should be understood, however, that any specific program names herein are used merely for convenience, and thus the invention should not be limited to use only with any specific application identified or implied or identified and implied by such names.

Claims

1. 1. A method for predicting excessive resource utilization in a distributed computing environment, comprising: obtaining, by one or more processors, at least a portion of code associated with a workload in a distributed computing environment; obtaining, by the one or more processors, accounting information associated with the workload; determining, by the one or more processors, a likelihood that the workload exhibits excessive resource utilization based at least in part on the at least a portion of the code and the accounting information associated with the workload; and Rescheduling, by the one or more processors, the workload in the distributed computing environment in response to a likelihood of excessive resource utilization exceeding a threshold. The method comprising:

2. 2. The method of claim 1, wherein rescheduling the workload in the distributed computing environment comprises one of terminating the workload, scheduling the workload in an isolated cluster of the distributed computing environment, scheduling the workload in a trusted cluster having one or more computing resource limitations, or restricting access to one or more resources of the distributed computing environment.

3. 3. The method of claim 1 or 2, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the at least one portion of code: a string pattern, a process pattern, or a file pattern.

4. 4. The method of claim 1, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the account information: account history, account age, account type, frequency of resource deployments, frequency of resource removals, previous quarantines, or other accounts with similar information to the account information associated with the workload.

5. 5. The method of claim 1, wherein the likelihood that the workload exhibits excessive resource usage is further based on one or more operational activities of the at least a portion of the code associated with a workload that matches a pattern of excessive resource usage.

6. 6. The method of claim 5, wherein the one or more operational activities of the at least one portion of code associated with a workload matching a pattern of excessive resource usage include one or more of a process usage pattern or a network usage pattern.

7. 7. The method of claim 1, wherein the likelihood that the workload is indicative of excessive resource usage is further based on a historical distribution of various points of previously executed workloads associated with other accounts in the distributed computing environment.

8. 1. A computer program for predicting excess resource utilization in a distributed computing environment, comprising: program instructions for obtaining at least a portion of code associated with a workload in a distributed computing environment; program instructions for obtaining accounting information associated with the workload; program instructions for determining a likelihood that the workload exhibits excessive resource utilization based at least in part on the at least a portion of the code and the accounting information associated with the workload; and program instructions for rescheduling the workload in the distributed computing environment in response to a likelihood of excessive resource utilization exceeding a threshold; The computer program product is executed by one or more processors.

9. 9. The computer program product of claim 8, wherein rescheduling the workload in the distributed computing environment comprises one of terminating the workload, scheduling the workload in an isolated cluster of the distributed computing environment, scheduling the workload in a trusted cluster having one or more computing resource limitations, or restricting access to one or more resources of the distributed computing environment.

10. 10. The computer program product of claim 8 or 9, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the at least one portion of code: a string pattern, a process pattern, or a file pattern.

11. 11. The computer program product of claim 8, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the account information: account history, account age, account type, frequency of resource deployment, frequency of resource removal, previous quarantine, or other accounts having similar information to the account information associated with the workload.

12. 12. The computer program product of claim 8, wherein the likelihood that the workload exhibits excessive resource usage is further based on one or more operational activities of the at least one portion of the code associated with a workload that matches a pattern of excessive resource usage.

13. 13. The computer program product of claim 12, wherein the one or more operational activities of the at least one portion of code associated with a workload matching a pattern of excessive resource usage include one or more of a process usage pattern or a network usage pattern.

14. 14. The computer program product of claim 8, wherein the likelihood that the workload exhibits excessive resource usage is further based on a historical distribution of various points of previously executed workloads associated with other accounts in the distributed computing environment.

15. 1. A computer system for predicting excessive resource utilization in a distributed computing environment, the computer system comprising: one or more computer processors; one or more computer-readable storage media; and a plurality of program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors; and a plurality of program instructions comprising: program instructions for obtaining at least a portion of code associated with a workload in a distributed computing environment; program instructions for obtaining accounting information associated with the workload; program instructions for determining a likelihood that the workload exhibits excessive resource utilization based at least in part on the at least a portion of the code and the accounting information associated with the workload; and program instructions for rescheduling the workload in the distributed computing environment in response to a likelihood of excessive resource utilization exceeding a threshold; Including, The computer system.

16. 16. The computer system of claim 15, wherein rescheduling the workload in the distributed computing environment comprises one of terminating the workload, scheduling the workload in an isolated cluster of the distributed computing environment, scheduling the workload in a trusted cluster having one or more computing resource limitations, or restricting access to one or more resources of the distributed computing environment.

17. 17. The computer system of claim 15 or 16, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the at least one portion of code: a string pattern, a process pattern, or a file pattern.

18. 18. The computer system of claim 15, wherein the likelihood that the workload exhibits excessive resource usage is based on one or more of the following aspects of the account information: account history, account age, account type, frequency of resource deployments, frequency of resource removals, previous quarantines, or other accounts with similar information to the account information associated with the workload.

19. 19. The computer system of claim 15, wherein the likelihood that the workload exhibits excessive resource usage is further based on one or more operational activities of the at least a portion of the code associated with a workload that matches a pattern of excessive resource usage.

20. 20. The computer system of claim 19, wherein the one or more operational activities of the at least one portion of code associated with a workload matching a pattern of excessive resource usage include one or more of a process usage pattern or a network usage pattern.

Images