Detecting Performance Degradation in Remotely Deployed Applications

Abstract

Techniques are provided for verifying performance degradation of a cloud-deployed application from neighborhood-based variability. Historical runs of an application deployed in a cloud environment are received. A subset of these historical runs is compared to performance metrics of a current version of the application deployed in the cloud environment using associated performance metrics recorded during the historical runs to determine a subset of historical runs that are similar to the current version. The determined subset is then used to compare performance metrics of a baseline run of the application to verify whether updating the application to the current version caused performance degradation, thereby reducing the impact of neighborhood-based variability on the assessment of performance degradation. Detailed performance analysis resources are less likely to be spent on performance degradation caused by neighborhood activity in a cloud environment.

Description

[Technical Field]

[0001] The present invention relates generally to the field of cloud-based software applications for detecting performance degradation in remotely deployed applications, and more particularly to isolating neighborhood-based variability from its impact on performance assessment for cloud-based software updates. [Background technology]

[0002] Cloud computing is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct, active management by the users of the resources. The term is commonly used to describe data centers available to many users via the Internet. Large-scale clouds, as is the case today, often distribute functions across several locations from a central server. Connections to users that are relatively close geographically can be designated as edge servers. Clouds can be limited to a single organization or can be available to multiple organizations, also known as multitenancy or multitenancy. Cloud computing relies on the sharing of resources to achieve consistency and increased value through economies of scale.

[0003] Software multitenancy refers to a software architecture in which a single instance of the software runs on a server and serves several tenants. Systems designed in this way are often called shared (as opposed to dedicated or isolated). A tenant is a user or group of users who share common, privileged access to a software instance. In a multitenant architecture, the software application is designed to provide each tenant with a dedicated share of the instance, including its data, user management, configuration, and tenant-specific functionality and non-functionality.

[0004] In software quality assurance, performance testing is generally a testing practice performed to measure how a system performs in terms of responsiveness and stability under a specific workload. It can also serve to investigate, measure, verify, or confirm other quality attributes of a system, such as scalability, reliability, and resource usage. Performance testing, a subset of performance engineering, is a computer science practice aimed at incorporating performance criteria into a system's implementation, design, and architecture. Performance testing can consist of running a specific version of software through "test runs," or functional exercises, and comparing responsiveness and resource usage compared to previous versions of the software. Summary of the Invention

[0005] According to one aspect of the present invention, there is provided a method, computer program product, or system, or combination thereof, that performs the following actions (not necessarily in the following order): (i) receiving a historical run data set corresponding to historical runs of a cloud application, (ii) receiving a monitoring data set corresponding to a current test run of an updated version of the cloud application, (iii) determining a subset of runs from the historical run data set that are most similar to the current test run, and (iv) automatically verifying that the current test run is actual performance degradation based at least in part on the determined subset of runs. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a block diagram of a first embodiment of a system according to the present invention; [Figure 2] 3 is a flow chart illustrating a method of the first embodiment that is at least partially performed by the system of the first embodiment. [Figure 3]FIG. 2 is a block diagram showing the machine logic (eg, software) portion of the system of the first embodiment. [Figure 4] FIG. 2 is a diagram of a screenshot generated by the first embodiment system. [Figure 5] 4 is a flow chart illustrating a method of the second embodiment. [Figure 6] 10 is a flow chart illustrating another aspect of the method of the second embodiment. [Figure 7] FIG. 1 illustrates a cloud computing environment according to one embodiment of the present invention. [Figure 8] FIG. 2 illustrates abstraction model layers according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0007] Some embodiments of the present invention relate to techniques for verifying performance degradation of cloud-deployed applications due to neighborhood-based variability. Historical runs of an application deployed in a cloud environment are received. In these embodiments, a subset of these historical runs is compared to performance metrics of a current version of the application deployed in the cloud environment using associated performance metrics recorded during the historical runs to determine a subset of historical runs that are similar to the current version. The determined subset is then used to compare performance metrics of a baseline run of the application to verify whether updating the application to the current version caused performance degradation, thereby reducing the impact of neighborhood-based variability on the assessment of performance degradation. Detailed performance analysis resources are less likely to be spent on performance degradation caused by neighborhood activity in the cloud environment.

[0008] This "Detailed Description" section is divided into the following subsections: (i) Hardware and Software Environment, (ii) Exemplary Embodiments, (iii) Further Comments and / or Embodiments, and (iv) Definitions.

[0009] I. Hardware and Software Environment The present invention may be a system, a method, or a computer program product, or a combination thereof, and may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0010] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0011] A "storage device" is defined herein as something fabricated or configured to store computer code so that the computer code can be accessed by a computer processor. A storage device generally includes a storage medium, which is the material on which the computer code data is stored. A single "storage device" may (i) have many separate, spaced or distributed parts (e.g., a set of six solid-state storage devices located in each of six laptop computers collectively storing a single computer program), or (ii) use multiple storage media (e.g., a set of computer code stored partially as a magnetic domain in the computer's nonvolatile storage and partially stored on a set of semiconductor switches in the computer's volatile memory), or both. The term "storage medium" should be interpreted to cover situations in which many different types of storage media are used.

[0012] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0013] The computer-readable program instructions for carrying out the operations of the present invention can be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or conventional procedural programming languages, such as object-oriented programming languages like Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0014] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0015] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, to produce a machine. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular way, such that the computer-readable storage medium on which the instructions are stored constitutes an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0016] The computer-readable program instructions may further be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed by the computer, other programmable apparatus, or other device, thereby resulting in a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device may perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0017] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware instructions and computer instructions.

[0018] As shown in FIG. 1, networked computer system 100 is one embodiment of a hardware and software environment for use with various embodiments of the present invention. Networked computer system 100 includes a server subsystem 102 (sometimes more simply referred to herein as subsystem 102), a cloud computing host computer 104, a client subsystem 110, and a communications network 114. Server subsystem 102 includes a server computer 200, a communications unit 202, a set of processors 204, a set of input / output (I / O) interfaces 206, memory 208, persistent storage 210, a display 212, external devices 214, random access memory (RAM) 230, a cache 232, and programs 300. Cloud computing host computer 104 includes a first cloud tenant 106 and a second cloud tenant 108.

[0019] Subsystem 102 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any other type of computer (see the definition of "computer" in the "Definitions" section below). Program 300 is a collection of machine-readable instructions and / or data used to create, manage, and control certain software functions discussed in detail below in the "Illustrative Embodiments" subsection of this "Detailed Description."

[0020] Subsystem 102 can communicate with other computer subsystems via communications network 114. Network 114 can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and can include wired, wireless, or fiber optic connections. In general, network 114 can be any combination of connections and protocols that support communication between server and client subsystems.

[0021] Subsystem 102 is shown as a block diagram with many double-headed arrows. These double-headed arrows (not separately numbered) represent a communications fabric that provides communication between the various components of subsystem 102. This communications fabric may be implemented using any architecture designed to pass data and / or control information between a processor (e.g., a microprocessor, a communications and network processor, etc.), system memory, peripheral devices, and any other hardware components within a computer system. For example, the communications fabric may be implemented, at least in part, using one or more buses.

[0022] Memory 208 and persistent storage 210 are computer-readable storage media. Generally, memory 208 can include any suitable volatile or non-volatile computer-readable storage media. It is further noted that, currently or in the near future, (i) external devices 214 may be able to provide some or all of the memory for subsystem 102, or (ii) devices outside of subsystem 102 may be able to provide memory for subsystem 102, or both. Both memory 208 and persistent storage 210 (i) store data in a manner less transient than a passing signal and (ii) store data in a tangible medium (e.g., magnetic or optical domain). In this embodiment, memory 208 is volatile storage, while persistent storage 210 comprises non-volatile storage. The medium used by persistent storage 210 may also be removable. For example, a removable hard drive may be used for persistent storage 210. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 210.

[0023] The communications unit 202 enables communication with other data processing systems or devices outside of the subsystem 102. In these examples, the communications unit 202 includes one or more network interface cards. The communications unit 202 may communicate via the use of either or both physical and wireless communications links. The software modules discussed herein may be downloaded to a persistent storage device (such as persistent storage 210) via a communications unit (such as the communications unit 202).

[0024] The I / O interface set 206 is in data communication with the server computer 200, enabling data input and output to and from other devices that may be locally connected. For example, the I / O interface set 206 provides a connection to an external device set 214. The external device set 214 will generally include devices such as a keyboard, keypad, touch screen, or any other suitable input device, or a combination thereof. The external device set 214 may further include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, such as the program 300, may be stored on such portable computer-readable storage media. The I / O interface set 206 is also in data communication with a display 212. The display 212 is a display device that provides a mechanism for displaying data to a user and may be, for example, a computer monitor or a smartphone display screen.

[0025] In this embodiment, program 300 is stored in persistent storage 210, typically via one or more memories in memory 208, for access and / or execution by one or more computer processors in processor set 204. Those skilled in the art will appreciate that program 300 may be stored in a highly distributed manner during execution, when not being executed, or both. Program 300 may include machine-readable, executable instructions, or actual data (i.e., the type of data stored in a database), or both. In this particular embodiment, persistent storage 210 includes a magnetic hard disk drive. Among other possible variations, persistent storage 210 may include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0026] The programs described herein are identified based on the application for which they are implemented in a particular embodiment of the invention. However, it should be recognized that specific program names herein are used merely for convenience, and thus the invention should not be limited to use only in the particular application identified and / or implied by such names.

[0027] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are chosen so as to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0028] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that practice of the teachings detailed herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0029] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0030] The features are as follows:

[0031] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the provider of the service.

[0032] Pervasive Network Access: Functionality is available across the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0033] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have no control or knowledge over the exact location of the resources provided, although there is a sense of location independence in that it may be possible to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0034] Rapid Elasticity: Capabilities can be rapidly and elastically provisioned and rapidly scaled out, and rapidly released and rapidly scaled in, sometimes automatically. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time.

[0035] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at an abstraction level appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0036] The service model is as follows:

[0037] Software as a Service (SaaS): The functionality offered to the consumer is the use of a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application features, with the possible exception of limited user-specific application configuration settings.

[0038] Platform as a Service (PaaS): The capability offered to the consumer is the deployment of consumer-created or acquired applications, created using programming languages and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0039] Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provisioning of processing, storage, network, and other basic computing resources, upon which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating system, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0040] The deployment model is as follows:

[0041] Private Cloud: The cloud infrastructure is operated solely for the organization. The cloud infrastructure may be managed by the organization or a third party and may reside on-premises or off-premises.

[0042] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organization or a third party and may reside on-premises or off-premises.

[0043] Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0044] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain distinct entities but are joined by standard or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0045] Cloud computing environments are service-oriented with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0046] Referring now to FIG. 7, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or combinations thereof, can communicate. The nodes 10 can communicate with each other. The nodes 10 may be physically or virtually grouped in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof (not shown). This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-54N shown in FIG. 7 are intended to be exemplary only, and that computing node 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0047] Referring now to Figure 8, a set of functional abstraction layers provided by cloud computing environment 50 (see Figure 7) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 8 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0048] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (reduced instruction set computer) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0049] The virtualization layer 70 provides an abstraction layer that can comprise the following examples of virtual entities: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0050] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides proactive provisioning and procurement of cloud computing resources in anticipation of future demand according to SLAs.

[0051] The workload tier 90 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and performance analyzer 96.

[0052] II. Illustrative Embodiments As shown in Figure 1, a networked computer system 100 is an environment in which an exemplary method in accordance with the present invention may be implemented. As shown in Figure 2, a flowchart 250 illustrates an exemplary method in accordance with the present invention. As shown in Figure 3, a program 300 performs or controls the execution of at least a portion of the operations of the method of flowchart 250. The method and associated software will now be discussed over the following paragraphs, with broad reference to the blocks of Figures 1, 2, and 3.

[0053] Processing begins at operation S255, where historical application run data store module (“mod”) 302 receives a historical application run data set. In this simplified embodiment, the application that is the subject of this example is called ExampleApp, and the newly updated version of ExampleApp is version 5. The historical application run data set includes three historical runs of ExampleApp (versions 1, 2, and 3), which include measurements of specific performance metrics for the operating periods of each of these versions. From version 1 through version 5, ExampleApp is, and has been, hosted on ExampleCloud as one of several tenants operating workloads on ExampleCloud. More specifically, in this simplified embodiment, ExampleCloud includes cloud computing host computer 104 of FIG. 1, with ExampleApp operating as first cloud tenant 106 and another tenant existing as second cloud tenant 108. In this example embodiment, the historical application run data set includes three performance metrics: (i) CPU cycles over the entire version run, (ii) input / output (IO) interface bits over the entire version run, and (iii) network interface upload / download bits over the entire version run. In the historical application run data set for ExampleApp, the performance metrics for Version 1 are: (i) 5,000 CPU cycles in 10 hours, (ii) 10,000 bits in 10 hours, and (iii) 3,000 bits in 10 hours. In the historical application run data set for ExampleApp, the performance metrics for Version 2 are: (i) 4,400 ticks in 8 hours, (ii) 8,040 bits in 8 hours, and (iii) 2,200 bits in 8 hours. In the historical application run data set for ExampleApp, the performance metrics for Version 3 are:(i) 6,300 ticks in 12 hours, (ii) 12,120 bits in 12 hours, and (iii) 3,180 bits in 12 hours. Additionally, the historical application run data set contains information corresponding to the most recent version, or "baseline run," of ExampleApp version 4, which includes the following performance metrics: (i) 2,850 CPU ticks in 6 hours, (ii) 5,970 bits in 6 hours, and (iii) 1,500 bits in 6 hours. The baseline version will be used as a basis for comparison in future operations.

[0054] In alternative embodiments, other metrics may be included in addition to or instead of those listed above, or there may be more runs corresponding to many different versions of the application. In another alternative embodiment, the performance metrics may correspond to even more fine-grained aspects of a given application, such as the CPU / IO / network performance of individual modules, services, microservices, or other such sub-portions of a given application.

[0055] Processing continues to operation S260, where the test run monitoring mod 306 monitors diagnostic information for the current test run of the updated application. In this simplified embodiment, the current test run is a test run of version 5 of ExampleApp, which has been operational and “running” for one hour. In this example embodiment, monitoring diagnostic information for version 5 of ExampleApp includes monitoring the following performance metrics: (i) CPU cycles over the entire version run, (ii) input / output (IO) interface bits over the entire version run, and (iii) network interface upload / download bits over the entire version run. In this example embodiment, the values measured during this run are: (i) 700 ticks per hour, (ii) 1100 bits per hour, and (iii) 460 bits per hour. In alternative embodiments, other metrics may be included in addition to or instead of those listed above. In another alternative embodiment, the performance metrics may correspond to even more fine-grained aspects of a given application, such as CPU / IO / network performance of individual modules, services, microservices, or other such sub-portions of a given application.

[0056] Processing continues at operation S265, where the data preprocessor mod 308 preprocesses the data from the monitored diagnostic information and the historical application runs. In this example embodiment, the monitored diagnostic information and the historical application runs are preprocessed to enable accurate comparisons in future operation of this embodiment. In this example embodiment, preprocessing involves averaging the performance metrics of the historical application run data set and the monitored diagnostic information of the current test run into hourly values. In this example embodiment, the monitored diagnostic information of the current test run has been measured over a one-hour period and includes the following performance metrics: (i) 700 ticks in one hour, (ii) 1100 bits in one hour, and (iii) 460 bits in one hour.

[0057] The performance metrics for version 1 of ExampleApp's historical application run data set are preprocessed as follows: (i) 5000 ticks in 10 hours are preprocessed by dividing 5000 ticks by 10 hours, which results in 500 ticks in 1 hour, (ii) 10000 bits in 10 hours are preprocessed by dividing 10000 bits by 10 hours, which results in 1000 bits in 1 hour, and (iii) 3000 bits in 10 hours are preprocessed by dividing 3000 bits by 10 hours, which results in 300 bits in 1 hour. The performance metrics for version 2 of ExampleApp's historical application run data set are preprocessed as follows: (i) 4400 ticks in 8 hours are preprocessed by dividing 4400 ticks by 8 hours, which results in 550 ticks in 1 hour, (ii) 8040 bits in 8 hours are preprocessed by dividing 8040 bits by 8 hours, which results in 1005 bits in 1 hour, and (iii) 2200 bits in 8 hours are preprocessed by dividing 2200 bits by 8 hours, which results in 275 bits in 1 hour. The performance metrics for version 3 of ExampleApp's historical application run dataset are preprocessed as follows: (i) 6300 ticks in 12 hours are preprocessed by dividing 6300 ticks by 12 hours, which results in 525 ticks in 1 hour, (ii) 12120 bits in 12 hours are preprocessed by dividing 12120 bits by 12 hours, which results in 1010 bits in 1 hour, and (iii) 3180 bits in 12 hours are preprocessed by dividing 3180 bits by 12 hours, which results in 265 bits in 1 hour. The performance metrics for version 4 of ExampleApp's historical application run dataset are preprocessed as follows:(i) 2850 ticks in 6 hours are preprocessed by dividing 2850 ticks by 6 hours, which gives 475 ticks in 1 hour; (ii) 5970 bits in 6 hours are preprocessed by dividing 5970 bits by 6 hours, which gives 995 bits in 1 hour; (iii) 1500 bits in 6 hours are preprocessed by dividing 1500 bits by 6 hours, which gives 250 bits in 1 hour.

[0058] In alternative embodiments, other preprocessing may be required based on differences in the underlying metrics. For example, a more granular application surrogate metric may have a metric corresponding to CPU usage spread across different submodules, which are then aggregated to define a performance metric corresponding to CPU usage for the entire application. Alternative preprocessing techniques may include eliminating irrelevant or unnecessary measurements or components detected by the monitoring module. In some alternative embodiments, the performance metrics are weighted by dynamically generated weights based on the average CPU cycles spent executing user application code (such as API calls), system OS / kernel functions, wait cycles waiting for I / O operations, and idle cycles where the CPU is not performing any tasks (corresponding to waiting for network inbound / outbound operations).

[0059] Processing continues at operation S270, where the subset determination modulus 310 determines a subset of runs that are most similar to the current test run. In this simplified embodiment, to determine the subset of runs, the subset determination modulus 310 determines three historical application runs (one for each of the three types of diagnostic information / performance metrics) as closest to the current test run based on three types of preprocessed diagnostic information / performance metrics: (i) CPU cycles over one hour, (ii) input / output (IO) interface bits over one hour, and (iii) network interface upload / download bits over one hour. The closest historical application run in terms of CPU cycles was the Version 2 run, which had 550 CPU ticks over one hour, which was only 150 ticks away from the CPU cycles at the time of the current test run. The closest historical application run in terms of IO interface bits was the Version 3 run, which had 1010 IO interface bits over one hour, which was only 90 ticks away from the IO interface bits at the time of the current test run. The closest historical application run for the network interface upload / download was the Version 1 run, which had 300 network interface uploads / downloads in one hour, which was only 160 bits away from the network interface upload / download at the time of the current test run. In an alternative embodiment, more than one historical application run is determined for each type of diagnostic information / performance metric, such that for a given type of diagnostic information / performance metric, several historical application runs are selected as most similar to the current test run. For example, five different historical application runs are determined to be closest to the current test run based on CPU cycles, and five other historical application runs are determined to be closest to the current test run based on IO interface bits.

[0060] Processing continues at operation S275, where the actual degradation determination modulus 312 determines whether the current test run exhibits actual performance degradation. In this simplified embodiment, actual performance degradation is determined by comparing the preprocessed performance metrics of the historical application runs of the determined most similar run with the preprocessed performance metrics of the baseline run. Actual performance degradation is determined if the comparison of at least one performance metric indicates that the value is greater than the baseline performance metric value above a threshold (10% in this illustrative embodiment). In this simplified embodiment, the baseline run is Version 4 of ExampleApp, which includes the following preprocessed performance metrics: (i) 475 CPU ticks in 1 hour, (ii) 995 IO interface bits in 1 hour, and (iii) 250 network interface upload / download bits in 1 hour. The determined most similar run in terms of CPU ticks was the Version 2 run, which included a now preprocessed CPU tick performance metric of 550 CPU ticks in 1 hour. This is compared to the baseline 475 CPU ticks for one hour by dividing 550 CPU ticks by 475 CPU ticks, resulting in a value of 1.16, or a 16% increase, above the baseline value, which exceeds the threshold for actual performance degradation and is therefore determined to be actual performance degradation. In an alternative embodiment, actual performance degradation may be determined if at least two performance metrics of the determined subset exceed the baseline performance metrics. In a further alternative embodiment, actual performance degradation is determined if three or more of the determined subset exceed the baseline performance metrics. For example, actual performance degradation is determined only if the comparisons between the baselines for CPU ticks, IO interface bits, and network interface upload / download bits all show values above a predetermined threshold.In a further alternative embodiment, where the determined subset for each performance metric is an average of the relevant performance metrics of that component's historical application runs, the determined averages are then summed together to determine an overall performance metric for the run that encapsulates various component performance metrics (e.g., CPU cycles, IO interfaces, network interfaces, etc.), which is then compared to a similar value derived from the baseline performance metrics.

[0061] In an alternative embodiment, the determined subsets for each diagnostic information / performance metric are averaged and weighted by the average distance between the diagnostic information of the current test run and the performance metrics of the various historical application runs in each subset that includes multiple historical application runs. For example, if there are three historical application runs determined to have CPU performance metrics most similar to the current test run, the CPU performance metrics are averaged and the average is weighted by the average distance between the CPU diagnostic information of the current test run and each of the CPU performance metrics.

[0062] Processing continues to operation S280, where flagging mod 314 flags version 5 of ExampleApp as having actual performance degradation. In this simplified embodiment, version 5 is flagged for further detailed performance analysis, and dialog 402 of screen 400 of Figure 4 is output to client 110 of Figure 1.

[0063] Processing continues at operation S285 of FIG. 2 , where analyzer mod 316 performs a detailed performance analysis of ExampleApp version 5. In this simplified embodiment, the computer program runs local copies of ExampleApp version 5 and ExampleApp version 4, measures performance metrics for various subcomponents of each version, and then outputs a table showing a side-by-side comparison of each performance metric and each subcomponent. In an alternative embodiment, a performance engineer digs into the code of ExampleApp version 5 to identify performance issues causing actual performance degradation. An improvement of embodiments of the present invention over the state of the art is the reduction of wasted resources spent on performance analysis of application versions or runs where suspected performance degradation is actually caused by workloads of “neighboring” cloud tenants running in the same cloud. By performing a comparison between the baseline and other historical application runs deemed most similar to the present application run, the influence of neighboring causes of performance anomalies is reduced in relation to determining whether detailed performance analysis is required.

[0064] III. Further Comments and / or Embodiments Some embodiments of the present invention recognize the following facts, potential problems, or potential areas for improvement over the current state of the art, or a combination thereof: (i) performance evaluation of a cloud-based product release involves running a performance workload on the build during the ongoing release cycle to compare it to a previous release (the previous release is also known as the baseline), (ii) performance degradation exceeding a threshold in the current release relative to the baseline is considered a degradation, (iii) this degradation often requires a performance engineer to conduct a thorough analysis to understand the cause of the degradation and determine how to optimize the code for better performance, and (iv) because the cloud is a multi-tenant environment, there are often performance fluctuations due to the activity of neighboring cloud tenants (where neighboring cloud tenants are workloads sourced from other parties running on the same hardware). (v) It is difficult to understand whether the degradation is due to code changes in the new release or to an unpredictable environment. (vi) For example, cloud-based Spark services run on virtual machines, and persistent storage is usually on object storage or S3, accessed over the cloud network. (vii) Developers often see spurious degradation when running performance tests on Spark services. (viii) This degradation still requires performance engineers to conduct thorough analysis to understand whether the regression is real or not. (xi) While there are ways to understand whether the environment is suitable for performance testing, there is no intelligent way to diagnose performance degradation and distinguish it from variability due to cloud proximity.

[0065] Some embodiments of the present invention may include one or more of the following operations, features, characteristics, or advantages, or a combination thereof: (i) during performance test runs, a monitoring tool is used to gain deep insights; (ii) based on these diagnostics, the three most similar past runs to the current performance test run are identified; (iii) these similar runs are then used to vote on whether the run in the test build is an actual regression or variability; (iv) toward achieving this, the K-Nearest Neighbor (KNN) algorithm appears to be a very natural fit; (v) KNN is a simple nonparametric algorithm that calculates a similarity measure (Euclidean distance) between a test sample and all supplied training samples; (vi) the K-Nearest Neighbors can then vote or predict the elapsed time of the test run weighted by the average distance from the test point; and (vii) distance is taken into account to understand how close or similar the test point is to other neighbors.

[0066] Some embodiments of the present invention utilize the following mechanism to weight the importance of the various diagnoses that contribute to elapsed time. Final elapsed time = W1*[Estimated elapsed time from CPU KNN cluster] + W2*[Estimated elapsed time from IO KNN cluster] + W3*[Estimated elapsed time from Network KNN cluster] where: W1 is the average of "user + system" CPU, W2 is the "waiting" CPU average, W3 is the factor of "idle" CPU and network bandwidth utilization.

[0067] Referring to the previous paragraph, if the calculated elapsed time of the baseline and the predicted elapsed time of the test run are within the threshold, the degradation is determined to be a false alarm. The weights attached to the diagnostics provide interpretability by indicating which aspects of the environment are causing the fluctuations. "User + System" refers to the total CPU time / cycles spent in user space (i.e., executing application program logic) and system space (i.e., executing kernel / OS functions). Similarly, "Wait" refers to the time spent waiting for IO operations, and "Idle" refers to when the CPU has no work to do. When an application is engaged in network activity, the CPU has nothing to do and is "Idle" during the network operation. If a program runs for one hour and CPU utilization (diagnostic) data is collected at one-minute intervals, 60 instances of diagnostic data will be generated. In each instance, CPU "User," "System," "Wait," and "Idle" values are generated, corresponding to the CPU cycles spent executing commands. The three weights mentioned above are determined as follows: (i) W1 is determined from the average of "User + System" from 60 instances of diagnostic data, (ii) W2 is determined from the average of "Wait" from 60 instances of diagnostic data, and (iii) W3 is determined from the average of "Idle" from 60 instances of diagnostic data.

[0068] Some embodiments of the present invention may include one or more of the following operations, features, characteristics, or advantages, or a combination thereof: (i) diagnostic data formats vary, so it is necessary to preprocess data that can be used in machine learning models; (ii) CPU diagnostic data preprocessing—diagnostics such as CPU profiler data or vmstat require some transformation before they can be fed to machine learning (ML) algorithms; (iii) this is because these diagnostics generate a large number of rows per performance run; and (iv) for example, CPU profiler captures all Db2 functions called during query execution, as shown below. Component, module (function) names will vary depending on the application workload and type.

[0069] [Table 1]

[0070] [Table 2]

[0071] To reduce dimensionality, this technique sums the CPU cycles spent by a function based on its components. It then creates a table with a column for each component_name and updates it so that each component_name generates one row per run. After this transformation, the data above becomes a table with 2 rows and 42 columns. The first row, the header row containing the corresponding component_names, contains the following columns: (1)PQA_RUN_ID, (2)pqa_test_id, (3)pqa_execution_id, (4)cde, (5)cde_data, (6)cde_query, (7)cde_services, (8)cde_types, (9)cli, (10)csm, (11)GLUE, (12)LIB_PTHREAD, (13)LIB_STDC, (14)pd, (15)sqk, (16)sqlb, (17)sqld, (18)sqle, (19)sqlf, (20)sq li, (21)sqlj, (22)sqlk, (23)sqlm, (24)sqln, (25)sqlnls, (26)sqlo, (27)sqlp, (28)sqlpg, (29)sqlpl, (30)sqlr, (31)sqlra, (32)sqlri, (33)sqlrihsjn, (34)sqlrl, (35)sqlrr, (36)sqlrs, (37)sqls, (38)sqlt, (39)sqlv, (40)sqlz, (41)stmm, and (42)UNKNOWN. The second row contains 42 columns corresponding to the total CPU cycles spent in the function corresponding to the associated component, excluding the first three columns corresponding to PQA_RUN_ID, pqa_test_id, and pqa_execution_id.(1)20449, (2)q20, (3)3, (4)NULL, (5)NULL, (6)749, (7)111, (8)121, (9)NULL, (10)NULL, (11)NULL, (12)NU LL, (13) NULL, (14) NULL, (15) NULL, (16) 147, (17) NULL, (18) NULL, (19) NULL, (20) 106, (21) NULL, (22) NULL, (23) NULL, (24) NULL, (25) NULL, (26) 157, (27) NULL, (28) NULL, (29) NULL, (30) NULL, (31) NULL, (32) NULL, (3 3)NULL, (34)NULL, (35)NULL, (36)NULL, (37)NULL, (38)NULL, (39)NULL, (40)NULL, (41)NULL, and (42)NULL.

[0072] To further reduce the dimensionality, columns that have a constant value for all rows are dropped (e.g., in this case, columns with NULL values). This can be achieved with the following command: X.drop([col for col, val in X.mean().iteritems() if val == 0], axis=1, inplace=True) The first row of the resulting table will contain only the following nine columns: (1) PQA_RUN_ID, (2) pqa_test_id, (3) pqa_execution_id, (4) cde_query, (5) cde_services, (6) cde_types, (7) sqlb, (8) 106, and (9) sqlo. The second row of the resulting table will contain only the following nine columns: (1) 20449, (2) q20, (3) 3, (4) 749, (5) 111, (6) 121, (7) 147, (8) 106, and (9) 157.

[0073] Some embodiments of the present invention utilize the Iostat tool to extract IO diagnostic data used in preprocessing. An example output based on Iostat is shown below. First, the average metric across the run is calculated for each device. Then, depending on whether the metric is a rate or a counter, the average or sum of the metric is calculated into a single column for each run, which is shown in the example table below with 16 columns and 5 rows. In general, iostat includes the following 16 metrics (columns): (1) Device, (2) r / s, (3) w / s, (4) rkB / s, (5) wkB / s, (6) rrqm / s, (7) wrqm / s, (8) %rrqm, (9) %wrqm, (10) r_await, (11) w_await, (12) aqu-sz, (13) rareq-sz, (14) wareq-sz, (15) svctm, and (16) %util. These metrics may vary based on the operating system used in the cloud infrastructure. The second row contains 16 columns: (1) loop0, (2) 0.00, (3) 0.00, (4) 0.00, (5) 0.00, (6) 0.00, (7) 0.00, (8) 0.00, (9) 0.00, (10) 0.00, (11) 0.00, (12) 0.00, (13) 1.60, (14) 0.00, (15) 0.00, and (16) 0.00. The third row contains 16 columns: The fourth row contains the following 16 columns: (1) vda, (2) 0.32, (3) 0.16, (4) 14.72, (5) 4.14, (6) 0.00, (7) 0.01, (8) 0.07, (9) 8.24, (10) 1.34, (11) 6.93, (12) 0.00, (13) 45.31, (14) 25.78, (15) 0.33, and (16) 0.02. The fifth and final row contains the following 16 columns: (1) dm-0, (2) 0.29, (3) 0.15, (4) 14.06, (5) 4.14, (6) 0.00, (7) 0.00, (8) 0.00, (9) 0.00, (10) 1.43, (11) 7.12, (12) 0.00, (13) 48.43, (14) 28.44, (15) 0.97, and (16) 0.04.(1) dm-1, (2) 0.01, (3) 0.00, (4) 0.20, (5) 0.00, (6) 0.00, (7) 0.00, (8) 0.00, (9) 0.00, (10) 1.17, (11) 0.00, (12) 0.00, (13) 22.87, (14) 0.00, (15) 0.96, and (16) 0.00.

[0074] Some embodiments of the present invention utilize the Netstat tool to extract network diagnostic data used in preprocessing. An example output based on Netstat is shown below. An example output based on Netstat is shown below. The metrics and actual column names may vary based on the operating system used in the cloud infrastructure. First, the average metric across the runs is calculated for each interface. Then, the sum of each metric is calculated to result in a single row per run, which is shown in the example kernel interface table below, with 11 columns and 4 rows. The first row, or header row, contains the following 11 columns: (1) Iface, (2) MTU, (3) RX-OK, (4) RXERR, (5) RX-DRP, (6) RX-OVR, (7) TX-OK, (8) TX-ERR, (9) TX-DRP, (10) TX-OVR, and (11) Flg. The second row contains the following 11 columns: The third row contains the following eleven columns: (1)ens3, (2)1500, (3)81835978, (4)0, (5)184, (6)0, (7)37, (8)0, (9)0, (10)0, and (11)BMRU. The third row contains the following eleven columns: (1)ens7, (2)1500, (3)53967, (4)0, (5)17, (6)0, (7)3265, (8)0, (9)0, (10)0, and (11)BMRU. The fourth and final row contains the following eleven columns: (1)lo, (2)65536, (3)16, (4)0, (5)0, (6)0, (7)16, (8)0, (9)0, (10)0, and (11)LRU.

[0075] Some embodiments of the present invention include methods illustrated in flowcharts 502, 514, and 526 of diagram 500 in Figure 5. Flowchart 502 includes the following steps, in the following order: (i) S504, (ii) S506, (iii) S508, (iv) S510, and (v) S512. Flowchart 514 includes the following steps, in the following order: (i) S516, (ii) S518, (iii) S520, (iv) S522, and (v) S524. Flowchart 526 includes the following steps, in the following order: (i) S528, (ii) S530, (iii) S532, (iv) S534, and (v) S536.

[0076] Some embodiments of the present invention include the method shown in flowchart 600 of Figure 6, which includes the following steps in the following order: (i) S602, (ii) S604, (iii) S606, (iv) S608, and (v) S610.

[0077] Some embodiments of the present invention may include one or more of the following operations, features, characteristics, or advantages, or combinations thereof: (i) determine whether performance degradation is due to changes made in the code path or due to noise in a completely unsupervised manner; (ii) do not require explicit training with labeled data; (iii) do not rely on artificial simulation to inject anomalies; (iv) leverage past runs to find runs that are closest to the current run with similar metrics and validate performance in a completely unsupervised / automated manner; (v) because the technique is unsupervised, the means of detection is robust and flexible to changes in the environment; (vi) use K-Nearest Neighbors (KNN), a simple non-parametric algorithm that calculates a similarity measure (Euclidean distance) between a test sample and all training samples; (vii) the K-Nearest Neighbors can then vote or predict the elapsed time of the test run; (viii) consider secondary metrics such as CPU, IO, and network utilization to estimate response times based on previously collected data; and (ix) this allows for a more accurate analysis of the code path of a transaction. (x) how to detect performance impact due to noisy neighborhoods or variations in a cloud environment; (xi) using a k-means clustering model to detect and flag performance variations in a cloud environment; (xii) determining performance variations without either (a) running the workload in a controlled environment or (b) aligning the behavior with another controlled test workload; (xiii) detecting performance variations and whether a workload has a true code path regression based on the normal execution of the workload in a cloud environment; (xiv) working on any cloud platform regardless of cloud architecture; (xv) detecting performance variations in a cloud environment and understanding whether an application code issue is affecting performance.

[0078] IV. Definition The present invention: This phrase should not be construed as an absolute indication that the subject matter described by the term "the present invention" is covered by the claims as filed or by the claims that may ultimately be issued after patent prosecution. The term "the present invention" is used to help the reader get a general sense that the disclosures herein are believed to be potentially novel, but this understanding implied by the use of the term "the present invention" is provisional and tentative, and is subject to change during patent prosecution as relevant information evolves and as the claims are potentially revised.

[0079] Embodiments: See the definition of "present invention" above; a similar caution applies to the term "embodiments."

[0080] or, ..., or both; or, ..., or combinations thereof: inclusive disjunction; for example, A, B, "or" C, "or combinations thereof" means that at least one of A or B or C is true and applicable.

[0081] Including / include / includes: Unless otherwise specified, means "including, but not necessarily limited to."

[0082] Module / Sub-Module: Any set of hardware, firmware, or software, or combination thereof, operable to function to perform some type of function, whether the module is (i) in a single local vicinity, (ii) distributed over a large area, (iii) in a single vicinity within a larger software code, (iv) located within a single software code, (v) located in a single storage device, memory, or medium, (vi) mechanically connected, (vii) electrically connected, or (viii) connected in data communication, or any combination thereof.

[0083] Computer: Any device capable of significant data processing and / or machine-readable instruction reading, including but not limited to desktop computers, mainframe computers, laptop computers, field-programmable gate array (FPGA)-based devices, smart phones, personal digital assistants (PDAs), body-worn or insertable computers, embedded device-style computers, and application-specific integrated circuit (ASIC)-based devices.

[0084] No Substantial Human Intervention: A process that occurs automatically (often through the operation of machine logic such as software) with little or no human input. Some examples that involve "no substantial human intervention" include (i) a computer performing a complex process and, due to a grid power outage, a human switches the computer to an alternate power source, allowing the process to continue uninterrupted; (ii) a computer is about to perform a resource-intensive process and a human confirms that the resource-intensive process should actually be undertaken (in this case, the confirmation process, considered in isolation, involves substantial human intervention, but the resource-intensive process does not involve any substantial human intervention, even though a simple yes-no style confirmation is required by a human); and (iii) a computer using machine logic makes a critical decision (e.g., a decision to ground all flights in anticipation of bad weather), but before implementing the critical decision, the computer must obtain a simple yes-no style confirmation from a human source.

[0085] Automatically: Without human intervention.

Claims

1. 1. A computer-implemented method executed by a computer system, the computer system comprising: receiving a historical run data set corresponding to a historical run of the cloud application; receiving a monitoring data set corresponding to a current test run of an updated version of the cloud application; determining a subset of runs from the historical runs data set that are most similar to the current test run; automatically verifying that the current test run is an actual performance degradation based at least in part on the determined subset of runs; and 10. A computer-implemented method comprising:

2. flagging the updated version of the cloud application for thorough performance analysis based at least in part on the automated validation; and analyzing the updated version of the cloud application for performance degradation in response to flagging the updated version of the cloud application; The computer-implemented method of claim 1 , further comprising:

3. 3. The computer-implemented method of claim 1, wherein the subset of runs that are most similar to the current test run is selected based at least in part on a K-nearest neighbor (KNN) clustering technique.

4. the subsets of runs include at least three distinct subsets of runs, with at least one subset of runs for each of the following three performance metrics: (i) central processing unit (CPU) usage, (ii) input / output (IO) interface usage, and (iii) network interface usage; The computer-implemented method of any one of claims 1 to 3, wherein the subsets of distinct runs each include at least two runs.

5. the three performance metrics are each weighted by a dynamically generated weight; the dynamically generated weights for the CPU usage performance metrics are based at least in part on an average of a total of CPU cycles used in executing machine logic corresponding to both user space and system space recorded in the subset of distinct runs; the dynamically generated weights for the IO interface usage performance metrics are based at least in part on averages of CPU wait cycles recorded for a subset of the distinct runs; 5. The computer-implemented method of claim 1, wherein the dynamically generated weights for the network interface usage performance metrics are based at least in part on an average of CPU idle cycles and a network interface bandwidth utilization rate.

6. The computer-implemented method of any one of claims 1 to 5, wherein the automated validation is based at least in part on a comparison of performance metrics of the determined subset of runs against a baseline run.

7. A computer program product for causing a computer system to execute the computer-implemented method according to any one of claims 1 to 6.

8. 1. A computer system comprising: receiving a historical run data set corresponding to a historical run of the cloud application; receiving a monitoring data set corresponding to a current test run of an updated version of the cloud application; determining a subset of runs from the historical runs data set that are most similar to the current test run; automatically verifying that the current test run is an actual performance degradation based at least in part on the determined subset of runs; and A computer system that executes the above.

9. flagging the updated version of the cloud application for thorough performance analysis based at least in part on the automated validation; and analyzing the updated version of the cloud application for performance degradation in response to flagging the updated version of the cloud application; 9. The computer system of claim 8, wherein the computer system executes:

10. 10. The computer system of claim 8 or 9, wherein the subset of runs that are most similar to the current test run is selected based at least in part on a K-Nearest Neighbor (KNN) clustering technique.

11. the subsets of runs include at least three distinct subsets of runs, with at least one subset of runs for each of the following three performance metrics: (i) central processing unit (CPU) usage, (ii) input / output (IO) interface usage, and (iii) network interface usage; The computer system of any one of claims 8 to 10, wherein the subsets of distinct runs each include at least two runs.

12. the three performance metrics are each weighted by a dynamically generated weight; the dynamically generated weights for the CPU usage performance metrics are based at least in part on an average of a total of CPU cycles used in executing machine logic corresponding to both user space and system space recorded in the subset of distinct runs; the dynamically generated weights for the IO interface usage performance metrics are based at least in part on averages of CPU wait cycles recorded for a subset of the distinct runs; 12. The computer system of claim 11, wherein the dynamically generated weights for the network interface usage performance metrics are based at least in part on an average of CPU idle cycles and a network interface bandwidth utilization rate.

13. The computer system of any one of claims 8 to 12, wherein the automated validation is based at least in part on a comparison of performance metrics of the determined subset of runs against a baseline run.

Images