Generate and update performance reports

Abstract

A computer-implemented method for generating and updating a performance report for a system adapted to execute multiple flows, allowing performance data of individual active flows to be compared with flows having similar functionality. The computer-implemented method is accomplished by linking each active flow to one of multiple templates representing the flow's functionality. Each template may include an associated performance summary generated for the template based on performance data of the active flows linked to the template. A performance report may be generated from the multiple performance summaries. The performance report may then be updated by executing multiple test flows at the system's spare capacity while obtaining test performance data from the test flows. The test performance data may be compared with performance data of the active flows on the system linked to the template.

Description

[Technical field]

[0001] The present disclosure relates to reporting performance of a system, and more particularly, to generating performance reports for a system configured to execute a plurality of flows. [Background technology]

[0002] Increasingly, customers are accessing applications provided by third parties that host said applications and deliver them to customers, typically over the Internet. This can occur in the form of software as a service (SaaS), infrastructure as a service (IaaS) or platform as a service (PaaS), or a combination of these. For many applications, customers purchase multiple licenses or processing power to be able to access the application.

[0003] The capacity of a system required by a customer is usually determined by a capacity planning process. Typically, the capacity planning process considers the type of processing the customer requires and then calculates the capacity required using sample performance data. However, because the scenarios need to be manually configured for testing (or configured using limited automation), the capacity planning process usually only involves sampling the performance of a very limited set of scenarios. In reality, there are many aspects that can vary, such as the complexity of the flow, the message size, and the number of backend calls required. With so many variations, obtaining an accurate forecast of capacity requirements is time-consuming and difficult. Thus, forecasts of required capacity often include 100% overhead to account for the inaccuracy of the forecast. In this case, the customer may end up paying twice the amount of capacity if the forecast turns out to be accurate. Summary of the Invention

[0004] According to an embodiment of the present disclosure, a computer-implemented method for generating a performance report for a system configured to execute a plurality of flows is provided. The method includes obtaining performance data associated with each of a plurality of active flows on the system. The method also includes obtaining a plurality of templates, each template representing a function of a flow. The method also includes linking each of the plurality of active flows to one of the plurality of templates, the template representing a function of the linked active flow. The method further includes generating, for each of the one or more templates, a performance summary based on the performance data associated with each of the plurality of active flows linked to the template. The method further includes generating a performance report based on the generated one or more performance summaries.

[0005] The concept of linking flows running on a system to templates that represent the flow's functionality is proposed. This allows for grouping of similar flows, whose performance can then be measured and compared. As a result, a performance report may be generated in which the performance of similar flows can be identified and compared. Thus, the proposed embodiments may provide context to performance data that is not otherwise available (e.g., average performance or range of possible performance for similar flows). Such context information may be useful to customers when evaluating performance or to inform changes made to the flows or the systems in which they run, or both.

[0006] By grouping the flows running on the system into templates, it may be possible to define multiple sets of similar flows. Performance data for all of the flows may then be collected and compared. In this manner, performance may be compared to other flows that have common functionality. By generating performance reports from the performance data, a customer can easily see how their flows compare to other similar flows, which may inform the customer's capacity planning process.

[0007] Thus, some proposed embodiments may involve generating a performance report that includes contextualized performance data for each of the flows running on the system. Linking of flows may be achieved by linking the flows to templates that represent the functionality of the flows. Many flows may be linked to each template, and when performance data is obtained for each flow, performance data for flows with similar functionality may be compared. A performance summary may be generated for each template, which may include, for example, the average performance, the best performance, and a list of all of the flows and their associated performance for each flow linked to the template.

[0008] Thus, some proposed embodiments may provide the advantage of being able to provide performance data that accurately reflects the actual performance characteristics of the system. In particular, accurate contextualized performance data may be provided, which can significantly improve the capacity planning process and help prevent various performance-related problems (e.g., resulting from insufficient capacity provision). Such information may be useful, especially for new customers planning to use the system, for example, by allowing the customer to understand as to how much capacity they require. As a further example, a customer may have flows that run without generating performance figures for the flows. Thus, a user may prefer to match flows against other customers' templates and corresponding performance figures.

[0009] Also, performance reports generated using data and templates from many customers should preferably prevent / minimize the leakage of information from any particular customer. Such information may relate, for example, to flow designs, details about message rates, or other business-sensitive information. To address this, embodiments may be configured to use templates that include corresponding performance figures within the report only if the data used to generate the report comes from two or more customers (or in some cases, more than a defined number of customers). In this way, the ability to infer a single customer's usage from the performance report may be reduced or prevented.

[0010] According to some embodiments, generating the performance summary may further include: determining, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined amount; and, in response to determining, for each of the one or more templates, that the number of active flows linked to the template satisfies the condition, generating a performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0011] Ensuring that each template contains more than a certain number of linked operational flows ensures that performance summaries are generated only when there is enough data to provide a meaningful comparison, thus reducing the impact of anomalous data.

[0012] According to some embodiments, obtaining performance data may include obtaining, for each of a plurality of active flows, one or more selected from the group consisting of: latency of the active flow, data rate per second of the active flow, cost of the active flow, and time the active flow ran on the system. Thus, the latency, data rate, and cost may be used to provide a measure of performance for each active flow. By recording the time the flow ran on the system, a performance summary may compare active flows occurring at a particular time, provide information about bottlenecks, and further inform the capacity planning process. It will be appreciated that other performance metrics may be used.

[0013] According to some embodiments, the performance summary may include one or more selected from the group consisting of an average value of the performance data of the multiple running flows linked to the template, a range of the performance data of the multiple running flows linked to the template, and a list of one or more of the performance data of the multiple running flows linked to the template.

[0014] By processing the data of all running flows linked to a template, further context may be given to the performance of individual running flows: for example, if the performance of a particular flow is much lower than the average performance of all of the running flows linked to the template, this may indicate a need for more capacity.

[0015] According to some embodiments, linking each of the plurality of active flows may include determining, for each of the plurality of active flows, a template from the plurality of templates that most closely represents a functionality of the active flow, and based on this determination, linking each of the plurality of active flows to one of the plurality of templates.

[0016] By ensuring that running flows are linked to templates that best represent the flow's functionality, data from each running flow can be compared to other similar running flows, which can provide meaningful context.

[0017] According to some embodiments, each of the multiple active flows may be associated with one of the multiple users, and each of the multiple users may be associated with one or more of the multiple active flows.

[0018] Each active flow may be associated with multiple users or customers, and the users of each active flow may be different, which may allow the performance of one user's active flow to be compared to similar active flows of various other users, each of which may have access to different capabilities of the system.

[0019] According to some embodiments, generating the performance report may further include receiving a query for the performance report from a user of the plurality of users; generating a performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on performance data associated with one or more of the plurality of running flows, where at least one of the one or more of the plurality of running flows is associated with the user; and in response to receiving the query for the performance report, transmitting the performance report to the user.

[0020] A user may request a performance report, which may be generated to display the performance of running flows associated with said user. Performance summaries that are not linked to running flows associated with a user may not be included, as they are irrelevant to said user.

[0021] According to some embodiments, the system may be a multitenancy integration cloud system, and each of a number of active flows may be assigned a default processing capacity of the system.

[0022] Each user running on the system may be assigned a certain amount of processing power of the system for executing running flows associated with the user. If a particular user is assigned low processing power, the performance of the running flows associated with the particular user may be worse than another user running similar running flows that is assigned a larger amount of processing power. This results in differences in the performance of running flows within a cloud-based multi-tenant system.

[0023] According to some embodiments of the present disclosure, a computer-implemented method for updating a performance report for a system configured to execute a plurality of flows is provided. The method includes obtaining a performance report, the performance report based on one or more performance summaries associated with one or more of a plurality of templates. The method also includes obtaining a plurality of test flows, each test flow associated with one of the plurality of templates. The method further includes executing a set of the plurality of test flows on the system. The method further includes obtaining test performance data associated with each of the set of the plurality of executed test flows. The method further includes generating, for at least one of the plurality of templates, a test performance summary based on the test performance data associated with each of the test flows associated with the template. The performance report is updated based on the generated one or more test performance summaries.

[0024] Another aspect of an embodiment of the present disclosure is the generation of additional performance data to provide further context to the performance data of flows linked to a template. This is accomplished by running test flows on a system, where the test flows are based on the template and are thus linked to flows running on the system linked to said template. In this manner, test performance data can be generated that may reflect, for example, the performance of a flow running at access to an ideal capacity level, and thus the best performance that the flow could realistically achieve. This data may then be provided to a user (e.g., by providing information regarding the performance that one or more flows may achieve) to further inform the capacity planning process.

[0025] It is proposed to further utilize templates linked to multiple flows to obtain multiple test flows. These test flows may then be executed on the system to provide more performance data in the form of test performance data and thus more context to the multiple flows linked to the template on which the test flows are based. The execution of the test flows may be used, for example, to update performance reports and thus provide more information to the user / customer and to inform the user / customer's capacity planning process.

[0026] According to some embodiments, obtaining the test performance data may include obtaining, for each of a set of a plurality of test flows, one or more selected from the group consisting of a latency of the test flow, a data rate per second of the test flow, a cost of the test flow, and a time that the test flow was executed on the system.

[0027] According to some embodiments, executing the set of test flows may include determining the set of test flows based on idle processing capacity of the system, and executing the set of test flows on the system.

[0028] This results in a selection of test flows that takes into account the spare capacity of the system, i.e., the spare capacity of the system utilized by a test flow can be maximized without affecting the parts of the system used by other flows. The test flows to be executed can be prioritized, for example, by which template the test flow was based on and executed most recently on the system, or by the number of flows linked to the template the test flow is based on.

[0029] According to some embodiments, determining the set of the plurality of test flows may further include determining an idle processing capacity of the system, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, increasing an amount of the system processing capacity used by the test flows, and in response to determining that the idle processing capacity of the system falls below the predefined capacity threshold, decreasing an amount of the system processing capacity used by the test flows.

[0030] In this way, test flows may be increased or decreased depending on the amount of system capacity being used by the actual live flows on the system. A reserve capacity overhead may be maintained to ensure sufficient capacity for the live flows.

[0031] According to some embodiments, increasing the amount of system processing capacity used by a test flow may include at least one selected from the group consisting of adding one or more test flows to the set of the plurality of test flows and increasing a data throughput of one or more of the test flows in the set of the plurality of test flows. According to some embodiments, decreasing the amount of system processing capacity used by a test flow may include at least one of removing one or more test flows from the set of the plurality of test flows and decreasing a data throughput of one or more of the test flows in the set of the plurality of test flows.

[0032] By adding test flows, performance data may be obtained for comparison with more operational flows. By increasing the data throughput, performance data of the test flows may be measured for comparison with operational flows having larger data throughput. Removal of test flows and reduction of data throughput may be performed to ensure sufficient processing power overhead.

[0033] Embodiments of the present disclosure also provide a method for generating a performance report for a system configured to execute a plurality of flows, according to some embodiments, and further including updating the performance report, according to other embodiments.

[0034] An embodiment of the present disclosure also provides a system comprising a processor and a computer-readable storage medium communicatively coupled to the processor and having program instructions stored thereon, which when executed by the processor, cause the processor to perform a method. The method performed by the processor includes obtaining performance data associated with each of a plurality of active flows on the system. The method also includes obtaining a plurality of templates, each template representing a function of the flow. The method also includes linking each of the plurality of active flows to one of the plurality of templates, the template representing a function of the linked active flow. The method further includes generating, for each of the one or more templates, a performance summary based on the performance data associated with each of the plurality of active flows linked to the template. The method further includes generating a performance report based on the generated one or more performance summaries.

[0035] According to some embodiments, generating the performance summary may further include: determining, by the processor, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined amount; and, in response to determining that, for each of the one or more templates, the number of active flows linked to the template satisfies the condition, generating a performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0036] According to some embodiments, obtaining the performance data may include obtaining, by the processor, for each of the plurality of running flows, one or more selected from the group consisting of: a latency of the running flow, a data rate per second of the running flow, a cost of the running flow, and an amount of time the running flow ran on the system.

[0037] According to some embodiments, the performance summary may include one or more selected from the group consisting of an average value of the performance data of the multiple running flows linked to the template, a range of the performance data of the multiple running flows linked to the template, and a list of one or more of the performance data of the multiple running flows linked to the template.

[0038] According to some embodiments, generating the performance report may further include receiving, by the processor, a query for the performance report from a user of the plurality of users; generating a performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on performance data associated with one or more of the plurality of running flows, where at least one of the one or more of the plurality of running flows is associated with the user; and transmitting the performance report to the user in response to receiving the query for the performance report.

[0039] According to some embodiments, the system may be a multi-tenant integrated cloud system, and each of a number of active flows may be assigned a default processing capacity of the system.

[0040] According to some embodiments of the present disclosure, a system is provided that includes a processor and a computer-readable storage medium communicatively coupled to the processor and having program instructions stored thereon, the program instructions, when executed by the processor, causing the processor to perform a method. The method performed by the processor includes obtaining a performance report, the performance report based on one or more performance summaries associated with one or more of a plurality of templates. The method also includes obtaining a plurality of test flows, each test flow associated with one of the plurality of templates. The method further includes executing a set of the plurality of test flows on the system. The method further includes obtaining test performance data associated with each of the plurality of sets of executed test flows. The method further includes generating, for at least one of the plurality of templates, a test performance summary based on the test performance data associated with each of the test flows associated with the template. The performance report is updated based on the generated one or more test performance summaries.

[0041] According to some embodiments, obtaining the test performance data may include obtaining, by the processor, for each of a set of a plurality of test flows, one or more selected from the group consisting of a latency of the test flow, a data rate per second of the test flow, a cost of the test flow, and a time the test flow was executed on the system.

[0042] According to some embodiments, executing the set of test flows may include determining, by a processor, the set of test flows based on idle processing capacity of the system, and executing the set of test flows on the system.

[0043] According to some embodiments, determining the set of multiple test flows may further include determining, by the processor, an idle processing capacity of the system; and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, increasing an amount of the system processing capacity used by the test flows; and in response to determining that the idle processing capacity of the system is below the predefined capacity threshold, decreasing an amount of the system processing capacity used by the test flows.

[0044] According to some embodiments, increasing the amount of system processing capacity used by a test flow may include at least one selected from the group consisting of adding one or more test flows to the set of the plurality of test flows and increasing a data throughput of one or more of the test flows in the set of the plurality of test flows. According to some embodiments, decreasing the amount of system processing capacity used by a test flow may include at least one of removing one or more test flows from the set of the plurality of test flows and decreasing a data throughput of one or more of the test flows in the set of the plurality of test flows.

[0045] According to some embodiments of the present disclosure, a computer program product is provided for generating a performance report for a system configured to execute a plurality of flows. The computer program product comprises a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to execute a method. The method includes obtaining performance data associated with each of a plurality of running flows executed on the system. The method also includes obtaining a plurality of templates, each template representing a function of a flow. The method also includes linking each of the plurality of running flows to one of the plurality of templates, the template representing a function of the linked running flow. The method further includes, for each of the one or more templates, generating a performance summary based on the performance data associated with each of the plurality of running flows linked to the template. The method further includes generating a performance report based on the generated one or more performance summaries.

[0046] The above summary is not intended to describe each example embodiment or every implementation of the present disclosure. These and other aspects of the present disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0047] For a better understanding of the present disclosure, and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]

[0048] [Figure 1] FIG. 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates an abstract model layer according to an embodiment of the present invention. [Figure 3A] FIG. 2 is a block diagram illustrating a set of three templates and example flows associated with each template according to an embodiment of the present disclosure. [Figure 3B] FIG. 2 is a block diagram illustrating three users and a set of example flows associated with each user, according to an embodiment of the present disclosure. [Figure 4] 1 is a flow diagram illustrating a method for generating a performance report according to an embodiment of the present disclosure. [Diagram 5] 1 is a flow diagram illustrating a method for updating a performance report according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating an apparatus configured to generate and update performance reports according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a high-level block diagram illustrating an example computer system that may be used in implementing one or more of the methods, tools, and modules described herein, and any associated functionality, in accordance with embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] The present disclosure is described with reference to the figures.

[0050] It should be understood that each figure is merely schematic and is not drawn to scale. It should also be understood that the same reference numbers are used throughout the figures to indicate the same or similar parts.

[0051] In the context of this application where the embodiments of the present disclosure constitute methods, it should be understood that such methods may be processes for execution by a computer, i.e. computer-implemented methods, and thus the various steps of the method may reflect various portions of a computer program (e.g. various portions of one or more algorithms).

[0052] Application and service integration typically involves (i) composing flows (otherwise called “application flows” or “integration flows”) that represent the flow of data between applications and the processing performed on data inputs and data outputs, and then (ii) parsing and transforming the data into a format that can be used by another application or service. Such flows are typically stored in formats such as eXtensible Markup Language (xml), Java(R)Script Object Notation (json), Ain't Markup Language (yaml), etc., and can be interpreted by a particular runtime as a set of instructions (each individual set of instructions is called a node) to communicate with other systems. A flow is composed by connecting together a set of nodes, each of which is adapted to perform a specific task (e.g., transform, filter, aggregate, etc.). An integration product, either on-premise or off-premise, then processes the data as specified by the flows. On-premise platforms are considered stable and provide a good level of security because data is stored and processed internally (e.g., within an internal private network). Off-premises platforms (e.g., cloud computing resources) are a relatively recent and evolving concept. References to off-premises resources or platforms are typically taken to refer to the concept of enabling ubiquitous, convenient, on-demand access via the Internet to a shared pool of configurable off-premises (e.g., remotely located) computing resources, such as networks, applications, servers, storage, and capabilities that are accessible via the Internet. Conversely, references to on-premises resources or platforms are taken to refer to the concept of local or private computing resources, such as networks, servers, storage devices, and applications, that are located inside / behind a local or virtual boundary (often behind a firewall).

[0053] According to a proposed embodiment, a method and / or apparatus is provided for generating a performance report for a system configured to execute a plurality of flows. Such a proposed method / apparatus obtains performance data associated with each of a plurality of active flows executed on the system. A plurality of templates are also obtained, each template representing a function of a flow. Then, each of the active flows is linked to a template representing a function of the linked active flow. Thereafter, for each of the one or more templates, a performance summary is generated based on the performance data associated with each of the plurality of active flows linked to the template. In other words, for each template, a performance summary is generated based on the ascertained performance of the flows linked to the template. Such a summary may represent the performance associated with a particular function. Then, a performance report may be generated based on the performance summary.

[0054] Linking flows running on a system to templates that represent the flow's functionality allows for grouping of flows that follow a particular template (i.e., related to the template's functionality). Performance data can be obtained for each flow running on the system that reflects performance. A performance summary can then be generated in which performance data for flows that follow a particular template can be compared and contrasted so that performance data collected for individual flows can be contextualized. This context can be useful to customers when evaluating performance and to inform changes made to the flows or the systems in which they run.

[0055] It is suggested that information overlooked by the customer during the capacity planning process may be related to the performance of the customer's flows and how this performance compares to the situation where the capacity is to be increased. By linking the flows running on the system to templates, multiple sets of similar flows can be obtained. Then, performance data on all of the aforementioned flows can be collected and the performance compared. In this way, the performance can be compared to other flows that have common functionality but may have access to, for example, different processing capacities of the system. Then, by generating a performance report, it is possible for the customer to easily see how the performance of a flow compares to other similar flows, which may inform the aforementioned customer's capacity planning process.

[0056] In some embodiments, generating the performance summary further includes: determining, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined amount; and, in response to determining, for each of the one or more templates, that the number of active flows linked to the template satisfies the condition, generating a performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0057] For example, this embodiment ensures that performance summaries are generated only if there are a sufficient number of running flows linked to them to ensure meaningful comparisons and to ensure that the impact of anomalous data is mitigated. The result is a performance summary that presents performance data that reflects the performance of a wide range of similar flows running in a variety of conditions.

[0058] According to some embodiments, obtaining the performance data includes, for each of the plurality of running flows, obtaining one or more of: a latency of the running flow, a data rate per second of the running flow, a cost of the running flow, and an amount of time the running flow ran on the system.

[0059] Templates linked to multiple flows can further be utilized to obtain multiple test flows. These test flows can then be executed on the system to provide more performance data in the form of test performance data and thus more context to the multiple flows linked to the template on which the test flows are based. The execution of the test flows can be used to update performance reports and thus provide more information to the user / customer and inform the user / customer's capacity planning process.

[0060] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the subject matter presented herein is not limited to a cloud computing environment. Embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0061] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computational resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) and for rapidly provisioning and releasing these resources with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0062] The features are as follows: On-Demand Self-Service: Cloud customers can automatically provision server time, network storage, and other computing power as they need it, without any unilateral, human interaction with the service provider. Broad network access: Cloud capabilities are available over the network and can be accessed using standard mechanisms, facilitating usage by heterogeneous thin- or thick-client platforms (e.g., mobile phones, laptops, and PDAs). Resource Pool: The provider's computing resources are pooled and offered to multiple consumers using a multi-tenant model. Different physical and virtual resources are dynamically allocated and reallocated according to demand. There is a sense of location independence, consumers usually have no control or knowledge regarding the exact location of the resources offered, although at a higher level of abstraction they may be able to specify a location (e.g. country, state, or data center). Rapid Elasticity: Cloud capacity can be provisioned quickly and elastically, in some cases automatically, to scale out quickly, and quickly released to scale in quickly. Capacity available for provisioning often appears to consumers as unlimited, available for purchase in any amount at any time. Metered Services: Cloud systems leverage metering to automatically control and optimize resource usage at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0063] The service model is as follows: SaaS (Software as a Service): The capability provided to the consumer is the use of the provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for the possibility of limited user-specific application configuration settings. PaaS (Platform as a Service): The capability provided to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to a consumer is the provisioning of processing, storage, network, and other basic computing resources over which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases has limited control over selected network components (e.g., host firewalls).

[0064] The deployment model is as follows: Private Cloud: This cloud infrastructure is operated solely for one organization. It can be managed by this organization or a third party and can exist on-premise or off-premise. Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It can be managed by these organizations or a third party and can reside on-premise or off-premise. Public Cloud: This cloud infrastructure is available for use by general users or large industry organizations and is owned by an organization that sells cloud services. Hybrid cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique but are bound together by standardized or proprietary technologies that allow for the portability of data and applications (e.g., cloud bursting to balance the load between clouds).

[0065] A cloud computing environment is a service-oriented environment with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that contains a network of interconnected nodes.

[0066] Referring now to FIG. 1, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers (e.g., a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N) can communicate. The nodes 10 may communicate with each other. The nodes 10 may be physically or virtually grouped in one or more networks (not shown), such as in a private cloud, community cloud, public cloud, or hybrid cloud, or combination thereof, as previously described herein. This allows the cloud computing environment 50 to provide an infrastructure, platform, and / or SaaS that does not require the cloud consumer to maintain resources on a local computing device. The types of computing devices 54A-N illustrated in FIG. 1 are intended as examples only, and it will be understood that the computing node 10 and cloud computing environment 50 may communicate with any type of computer-controlled device via any type of network and / or network-addressable connection (e.g., a connection using a web browser).

[0067] Referring now to Figure 2, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 1) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 2 are intended to be illustrative only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0068] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, Reduced Instruction Set Computer (RISC) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0069] The virtualization layer 70 provides an abstraction layer that can provide virtual entities such as virtual servers 71 , virtual storage 72 , virtual networks including virtual private networks 73 , virtual applications and operating systems 74 , and virtual clients 75 .

[0070] As an example, the management layer 80 may provide the functions described below: Resource provisioning 81 provides dynamic procurement of computational and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the utilization of those resources. As an example, those resources may include application software licenses. Security provides identity verification of cloud users and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment to users and system administrators. Service level management 84 provides allocation and management of cloud computational resources to meet required service levels. Performance reporting 85 provides generation of performance reports according to the proposed concepts detailed herein.

[0071] The Workload Layer 90 illustrates examples of functionality available in a cloud computing environment. Examples of workloads and functionality that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and time-bound virtual currency processing 96.

[0072] Example embodiments may be utilized in various types of distributed processing environments. To provide a context for the description of the elements and functionality of example embodiments, the following diagrams are provided as example environments in which aspects of example embodiments may be implemented. It should be understood that these diagrams are merely examples, and are not intended to assert or imply any limitation with respect to the environments in which aspects or embodiments of the invention may be implemented. Many modifications to the depicted environments may be made without departing from the spirit and scope of the invention. Additionally, an apparatus or system embodying one or more of the proposed concepts may take the form of any of a number of different processing devices, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communications devices, personal digital assistants (PDAs), and the like. In some examples, off-premise devices and on-premise devices may include portable computing devices configured using flash memory to provide non-volatile memory, for example, to store operating system files and / or user-generated data. Thus, the system may essentially be any known or later developed processing system, without architectural limitations.

[0073] The proposed concept may improve systems by providing an apparatus or method for generating performance reports for systems running multiple flows. Such proposals may enhance or improve capacity planning and / or allocation capabilities.

[0074] To aid in understanding the proposed concept, an example embodiment will now be described with reference to Figures 3A and 3B.

[0075] FIG. 3A illustrates a block diagram 310 showing three templates and an example set of flows 318 associated with each template, according to an embodiment of the present disclosure.

[0076] A flow may be thought of as a set of operations connected together to accomplish a task. For example, a flow can be an integration where an input to one system or application is connected to a call to another system. A running flow may be a flow that is running or has been run on a system configured to run multiple flows.

[0077] A template may represent a pattern of one flow, or may represent a pattern of more than one flow. A flow may be created from a template. In cases where a template represents a pattern of more than one flow, there may be an element of variability in the pattern that the template represents, such that more than one flow may be created from the template. A set of templates may be generated from a set of flows, such that, for example, 100 flows generate 100 templates, or 100 flows generate 50 templates, or 100 flows generate 10 templates.

[0078] As shown in FIG. 3A, template number 1 312 may best represent the pattern or function of flow number 2, flow number 5, flow number 6, and flow number 8. Template number 2 314 may best represent the pattern or function of flow number 1, flow number 3, and flow number 7. Template number 3 316 may best represent the pattern or function of flow number 4. If there is a flow with a pattern or function that cannot be represented by any of the existing templates, a new template may be generated from the aforementioned flows. If there is a flow with a pattern or function that cannot be represented by any of the existing templates, a new template may be generated from the aforementioned flows and one or more other flows. If there is a flow with a pattern or function that cannot be represented by any of the existing templates, a new template may be generated by generalizing an existing template.

[0079] An active flow may be linked to a template that represents a function or pattern of the active flow. An active flow may be linked to a template that best represents the function or pattern of the active flow. The template that best represents the function or pattern of the active flow may be determined by a measure of variability in the pattern of flow represented by the template and the actual pattern of the active flow.

[0080] FIG. 3B illustrates a block diagram 320 illustrating three users and an example set of flows 328 associated with each user, according to an embodiment of the disclosure.

[0081] The system may execute multiple active flows. Each of the multiple active flows may be associated with a user. Each user may have multiple active flows. As shown in FIG. 3B, user number 1 322 is associated with active flow number 5 and flow number 6. User number 2 324 is associated with active flow number 1, flow number 2, flow number 3, and flow number 4. User number 3 326 is associated with active flow number 7 and flow number 8. The active flows associated with one user may have some similarity with the active flows associated with another user such that the flows are linked to the same template.

[0082] FIG. 4 illustrates a flow diagram 400 illustrating a method for generating a performance report according to an embodiment of the present disclosure.

[0083] At step 402, performance data is obtained from a number of running flows, which may include, but are not limited to, measuring the latency of the running flows, measuring the data rate of the running flows, measuring the system cost in terms of memory for the running flows, and measuring the system cost in terms of processing the running flows. The time that the running flows have been running on the system may be recorded.

[0084] At step 404, a number of templates are obtained. These templates may be obtained from a template database. Alternatively, the templates may be generated from a number of running flows. The template may represent a function of a flow, or may represent one or more functions of the number of running flows.

[0085] At step 406, each of the multiple active flows is linked to a template that represents the functionality of the active flow. The template to which the active flow is linked may represent the active flow. A best match with the pattern of the active flow may be determined by minimizing the variability in the pattern of flows represented by the templates. Alternatively, the active flow may be linked to a template that represents the fewest number of functions of the flow, but still represents (or most closely represents) the functionality of the active flow.

[0086] A live flow may be linked to only one of the multiple templates, which provides the advantage that it is linked only to the template that best represents the functionality of the live flow. Alternatively, a live flow may be linked to more than one of the multiple templates, which provides the advantage that it is compared to a wider range of different flows so that there are more performance data points for comparison.

[0087] At step 408, a performance summary is generated for each template of the plurality of templates. If the number of active flows linked to the template is below a certain number, a performance summary may not be generated for the template. For example, if only one active flow is linked to the template, a performance summary may not be generated. This ensures that each generated performance summary includes performance data for a sufficient number of active flows so that anomalous data can be corrected.

[0088] Generating a performance summary for a particular template may include, but is not limited to, calculating an average value of all the performance data of the running flows linked to the template, calculating a range of all the performance data of the running flows linked to the template, ordering a list of all the performance data of the running flows linked to the template, and / or calculating the maximum and minimum values ​​of all the performance data of the running flows linked to the template.

[0089] For example, a performance summary may be generated using performance data from running flows at a particular time, such that the performance summary is specific to performance data from flows that ran on the system during the morning, afternoon, a particular time of day, or a particular day of the week. Performance data from flows running at different times may be compared and contrasted in a performance summary to assist the user in identifying times of bottlenecks or excess capacity. This aids in the capacity planning process and also aids in the planning process of when particular flows should be run on the system.

[0090] At step 410, a performance report is generated, which may be based on one or more of the performance summaries generated at step 408.

[0091] In some embodiments, a performance report query is received from a user. The user may be associated with multiple running flows on the system that are linked to multiple templates. Upon receiving the performance report query, a performance report is generated. In this case, the performance report may be generated based on one or more performance summaries, each of which is associated with a template linked to one or more running flows associated with said user. This means that only performance summaries relevant to the user are presented to the user in the performance report. Upon generation of the performance report, the performance report is sent to the user.

[0092] In one example, a user may select which performance data to include in the performance report and how the performance data is processed to generate each of the performance summaries on which the performance report is based. In another example, the performance data and how the performance data is used to generate each of the performance summaries on which the performance report is based are determined automatically.

[0093] For example, a user may be associated with two active flows. Each of the active flows may be linked to a respective template. Performance data for each active flow is obtained and then compiled into a performance summary that includes performance data of the other active flows linked to the template. The performance summary is then used to generate a performance report. The performance report is then sent to the user. The performance report may include performance data of the two active flows associated with the user, an average value of the performance data of other flows similar to the two active flows, and the best performance data of other flows similar to the two active flows. Additionally, performance data of each active flow at different times may be presented in the performance report.

[0094] In other words, the generation of the performance report can be described as follows: Each user's running flows are matched with a template that best represents the functionality of the running flows. Virtually all running flows are classified into groups of flows that follow a particular template. Then, the performance data collected from the running flows for the corresponding template is stored in the form of a performance summary. This storage may be performed only if a certain number of different running flows are linked to the same template, in order to anonymize the performance data. Then, the performance summary may be collected and added to an automatically generated performance report, which shows the average performance indicators for each template.

[0095] FIG. 5 illustrates a flow diagram 500 illustrating a method for updating a performance report according to an embodiment of the present disclosure.

[0096] At step 502, a performance report is obtained that is related to the performance of a plurality of active flows. The performance report may be based on one or more performance summaries, each of which may be associated with a template. Each of such templates may be linked to a plurality of active flows.

[0097] At step 504, a plurality of test flows are obtained, and each of the test flows is associated with a template. There may be at least one test flow for each template associated with each performance summary in the performance report. Each test flow may be generated from said template. Alternatively, each test flow may be obtained from a database of test flows. Each test flow may share common functionality with one or more of the plurality of operational flows.

[0098] In step 506, a set of multiple test flows is executed and test performance data is obtained for each test flow.

[0099] The set of multiple test flows is determined based on the spare capacity of the system. This capacity may be, for example, processing power or storage capacity. The amount of capacity that the test flows use is determined by the spare capacity of the system.

[0100] If the spare capacity of the system exceeds a particular amount, percentage, or threshold, the number of test flows on the system may be increased. Alternatively, if the spare capacity of the system exceeds a particular amount, percentage, or threshold, the data throughput or drive rate of test flows already running on the system may be increased.

[0101] If the spare capacity of the system falls below a particular amount, percentage, or threshold, the number of test flows on the system may be reduced. Alternatively, if the spare capacity of the system falls below a particular amount, percentage, or threshold, the data throughput or drive rate of test flows already running on the system may be reduced.

[0102] In this way, the capacity of the system available for the operational flows is kept at an acceptable level to account for variations in the load of the operational flows. The available capacity of the system may be continuously monitored and the set of test flows continuously re-evaluated to ensure acceptable available capacity of the system.

[0103] The multiple test flows in the set of test flows selected to run on the system may be selected based on, for example, the last time the test flow was run, the last time a test flow associated with a particular template was run, or the number of running flows that have a similar pattern or functionality to the test flow, or any combination thereof. In this manner, a variety of test flows may be run such that test performance data associated with a large number of running flows is obtained.

[0104] Test performance data for each test flow is obtained while each of the test flows is executed on the system, which may include, but is not limited to, measuring the latency of the test flow, measuring the data rate of the test flow, measuring the system cost in terms of memory for the test flow, and measuring the system cost in terms of processing the test flow. The time that the test flow is running on the system may be recorded.

[0105] At step 508, a test performance summary is generated. Based on the test performance data obtained in the previous step, the test performance summary is generated. Each test performance summary is associated with one of a plurality of templates associated with a performance report.

[0106] Generating a test performance summary for a particular template may include, but is not limited to, calculating an average value of all test performance data of the Test Flows linked to the template, calculating a range of all test performance data of the Test Flows linked to the template, ordering a list of all test performance data of the Test Flows linked to the template, and / or calculating maximum and minimum values ​​of all test performance data of the Test Flows linked to the template.

[0107] For example, a test performance summary may be generated using test performance data from test flows at a particular time, such that the test performance summary is specific to performance data from flows that ran on the system during the morning, afternoon, a particular time of day, or a particular day of the week. Test performance data from test flows running at different times may be compared and contrasted in the test performance summary to assist the user in identifying times of bottlenecks or excess capacity. This can aid in the capacity planning process and also in the planning process of when a batch of flows should be run on a system, or when a flow should be run so that performance exceeds a particular level.

[0108] At step 510, the performance report is updated. The test performance summary is added to the performance report. The test performance summary associated with the template associated with the performance summary in the performance report is added. In this way, test performance data that is relevant to the running flows contained in the performance report is added. This may give clues to the user as to the likely cost, latency, and maximum throughput for the user's running flows, since the user can compare the performance of their running flows with the performance of other running flows on the system from the original performance report, and with the performance of the test flows.

[0109] In other words, the update of the performance report can be stated as follows: For each identified template, a test flow can be created manually in the same way that flows are normally created from templates. In practice, the tests run on the system may be based on a subset of the templates. This subset can be determined based on the number of running flows associated with each template, and also based on how different the templates are from the other templates. The subset of test flows may then be run on the system in successive cycles, with the drive speed being increased until a certain percentage of the total processing capacity of the system is reached. After exceeding a threshold, the test flows are "ramp-backed" to keep the processing available for the running flows at an acceptable level. The test performance data from the test flows is then combined with the numerical performance data of the running flows in the performance report. This allows the report to show, for a particular template, the rates achieved by actual users running the flows, and to show the maximum rate that may be achieved by the test flows. From this, the user may be informed of the possible costs, latencies, and maximum throughput. It can be expected that there will be many fewer data points of test performance data from a test flow than performance data from a production flow.

[0110] The performance report may be updated periodically in the manner described above, for example each time a performance report is generated, each time a user submits a performance report query, or each time a performance summary is generated.

[0111] FIG. 6 illustrates a block diagram 600 illustrating an apparatus for generating and updating performance reports according to an embodiment of the present disclosure.

[0112] The system 610 is configured to execute multiple flows, including multiple running flows 612 and multiple test flows 614. The system 610 may be a cloud-based, multi-tenant system configured to execute multiple flows from multiple users, each of which may have access to certain processing power of the system. The system 610 may be a cloud-based, integrated product, such as a cloud-based application, examples of which are known and widely available.

[0113] The mapping unit 620 may be configured to store the templates in a template store 622. The mapping unit 620 may be configured to store links between the running and test flows and the templates in a template-flow link store 624. The mapping unit 620 is configured to retrieve a number of templates and link each of the running flows to one or more of the aforementioned templates. The mapping unit 620 may be configured to determine the template that best represents a function or pattern of the running flow.

[0114] The index unit 630 is configured to obtain performance data associated with each of the plurality of active flows 612 on the system 610. The index unit 630 may be further configured to store the performance data associated with each of the plurality of active flows on the system 612 in a performance data store 632.

[0115] The index unit 630 is also configured to obtain test performance data associated with each of the plurality of test flows 614 on the system 610. The index unit 630 may be further configured to store the test performance data associated with each of the plurality of test flows 614 on the system 610 in a test performance data store 634.

[0116] The index unit 630 may be configured to obtain the latency, data rate, cost, and execution time of the flows running on the system 610.

[0117] A performance test driver 640 is configured to obtain a plurality of test flows, determine a set of test flows to be executed on the system based on the spare processing capacity of the system 610, and drive the set of the plurality of test flows 614 on the system 610. The performance test driver 640 may be configured to store the test flows in a test flow store 642. The performance test driver 640 may further be configured to determine an idle capacity of the system 610, and increase or decrease a capacity of the system 610 used by the test flows 614. The performance driver 640 may further be configured to add or remove test flows to or from the set of test flows 614 on the system 610, and increase or decrease a data throughput of the set of the plurality of test flows 614 on the system 610.

[0118] The report generator 650 is configured to generate a performance summary and store the performance summary in the performance summary store 652. The report generator 650 may be configured to obtain, for each template, from the mapping unit 620, which active flows are associated with the template, and then aggregate performance data from the index unit 630 associated with the active flows into a performance summary. Aggregating the performance data may include averaging the performance data, determining a range for the performance data, or determining maximum and minimum values ​​for the performance data, or performing a combination thereof. The performance report generator 650 may not generate a performance summary for a template if less than a certain number of active flows are linked to the template.

[0119] The report generator 650 is further configured to generate a test performance summary and store the test performance summary in a test performance summary store 654. The report generator 650 may be configured to obtain which test flows are associated with the template from the mapping unit 620 and then summarize the test performance data from the index unit 632 associated with the test flows into a test performance summary. Summarizing the test performance data may include averaging the test performance data, determining a range for the test performance data, or determining maximum and minimum values ​​for the test performance data, or a combination thereof.

[0120] The report generator 650 is further configured to generate and update the performance reports. The report generator 650 is configured to store the performance reports in a performance report store 656.

[0121] The report generator 650 may generate a performance report based on the number of performance summaries. The report generator 650 may be configured to receive a query for a performance report from a user, generate a performance report based on the performance summaries linked to the user via a running flow linked to a template with which the performance summaries are associated, and send the performance report to the user. Alternatively, the performance report generator 650 may be further configured to update the performance report with the test performance summaries before sending the performance report to the user. The report generator 650 may be configured to store the performance report and to continuously update the performance report. Alternatively, the report generator 650 may be configured to store the performance report and to update the performance report when a query for a performance report is received from a user.

[0122] 7, there is shown a high-level block diagram of an exemplary computer system 1101 that may be used in implementing one or more of the methods, tools, and modules described herein, and any associated functionality (e.g., using one or more processor circuits or computer processors of a computer), in accordance with an embodiment of the present disclosure. In some embodiments, the major components of the computer system 1101 may include one or more CPUs 1102, a memory subsystem 1104, a terminal interface 1112, a storage interface 1116, an I / O (Input / Output) device interface 1114, and a network interface 1118, all of which may be communicatively coupled, directly or indirectly, for communication between the components via a memory bus 1103, an I / O bus 1108, and an I / O bus interface 1110.

[0123] The computer system 1101 may include one or more programmable general-purpose central processing units (CPUs) 1102A, 1102B, 1102C, and 1102D, collectively referred to herein as CPUs 1102. In some embodiments, the computer system 1101 may include multiple processors, as is typical for larger systems, while in other embodiments, the computer system 1101 may alternatively be a single-CPU system. Each CPU 1102 may execute instructions stored in a memory subsystem 1104 and may include one or more levels of on-board cache. In some embodiments, the processors may include at least one or more of a memory controller and / or storage controller. In some embodiments, the CPUs may execute the processes contained herein (e.g., processes 400 and 500).

[0124] The system memory subsystem 1104 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1122 or cache memory 1124. The computer system 1101 may further include other removable / non-removable, volatile / non-volatile computer system data storage media. By way of example only, a storage system 1126 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium, such as a "hard drive." Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), or an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In addition, the memory subsystem 1104 may include flash memory (e.g., a flash memory stick drive or flash drive). The memory devices may be connected to the memory bus 1103 by one or more data media interfaces. The memory subsystem 1104 may include at least one program product comprising a series of (eg, at least one) program modules configured to perform the functions of various embodiments.

[0125] 7 is depicted as a single bus structure providing a direct communication path between CPU 1102, memory subsystem 1104, and I / O bus interface 1110, memory bus 1103 in some embodiments may include multiple different buses or communication paths, which may be arranged in any of a variety of configurations, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, redundant parallel paths, or any other suitable type of configuration. Additionally, while I / O bus interface 1110 and I / O bus 1108 are depicted as a single unit, computer system 1101 in some embodiments may include multiple I / O bus interfaces 1110, multiple I / O buses 1108, or both. Additionally, although multiple I / O interface units are shown isolating the I / O bus 1108 from the various communication paths reaching the various I / O devices, in other embodiments some or all of the I / O devices may be directly connected to one or more system I / O buses.

[0126] In some embodiments, computer system 1101 may be a multi-user mainframe computer system, a single-user system, or a server computer, or similar device that has little or no direct user interface but receives requests from other computer systems (clients). Further, in some embodiments, computer system 1101 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, a network switch or router, or any other suitable type of electronic device.

[0127] It is noted that Figure 7 is intended to illustrate major representative components of an exemplary computer system 1101. However, in some embodiments, individual components may have greater or less complexity than those depicted in Figure 7, there may be other or additional components than those depicted in Figure 7, and the number, type, and configuration of such components may vary.

[0128] One or more programs / utilities 1128, each including at least one set of program modules 1130, may be stored in the memory subsystem 1104. The programs / utilities 1128 may include a hypervisor (also called a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or a combination thereof, may include an implementation of a network environment. The programs / utilities 1128 and / or program modules 1130 typically perform the functions or methods of the various embodiments.

[0129] From the above description, it will be appreciated that the proposed embodiments may provide one or more concepts for generating performance reports using actual production systems on which end users perform integrations. The example embodiments may be described as including two main parts:

[0130] (i) collecting performance results from actual customers running the flows and allocating the results to specific integration types (i.e., flow functions) within the performance reports; and

[0131] (ii) A cycle repeated by running performance tests for each type of integration using spare capacity in the production system, with automatic reduction after capacity reaches a certain level.

[0132] Results from the tests are combined with user / customer figures, allowing reports to show speeds achieved by actual customers for a particular function, or maximum values ​​possible based on manually created tests, or both. From such information, the user may know, for example, possible costs, latencies, and maximum throughput. This may allow the user to evaluate planned usage and begin to adjust, modify, or match flows according to requirements. Furthermore, results may be recorded based on message size and time of day to provide the customer with additional information on how the performance of a particular type of flow changes throughout a period of time (e.g., a 24-hour period), thus allowing improved planning.

[0133] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium including computer readable program instructions for causing a processor to perform aspects of the present invention.

[0134] A computer readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable floppy disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0135] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing device / processing device or to an external computer or storage device over a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof) that may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing device / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage on a computer readable storage medium within each computing device / processing device.

[0136] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the invention, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to customize the electronic circuitry by utilizing state information of the computer readable program instructions.

[0137] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks included in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0138] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams.

[0139] Computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in the blocks of the flowcharts and / or block diagrams, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process.

[0140] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than the order shown in the figures. For example, two blocks shown in succession may in fact be executed as one step, executed simultaneously, executed substantially simultaneously in a partially or fully overlapping manner in time, or executed in reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flowchart diagrams, as well as combinations of blocks included in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0141] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many changes and modifications that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

[0142] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the various embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context expressly indicates otherwise. It will be further understood that the terms "comprise" and / or "comprising", when used herein, indicate the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof. In the preceding detailed description of exemplary embodiments of the various embodiments, reference is made to the accompanying drawings, which form a part hereof (where like numbers represent like elements), in which specific example embodiments are shown by way of example and in which the various embodiments may be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical and other changes may be made without departing from the scope of the various embodiments. In the preceding description, many specific details have been set forth to enable a thorough understanding of the various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail so as not to obscure the embodiments.

[0143] As used herein, "plurality," when used in reference to an item, means one or more of the item. For example, "plurality of different types of networks" is one or more different types of networks.

[0144] Where different reference numerals include a common number followed by a different letter (e.g., 100a, 100b, 100c) or followed by a separator character followed by a different number (e.g., 100-1, 100-2, or 100.1, 100.2), use of the reference letter alone (e.g., 100) without a letter or following number may refer to the entire group of elements, any subset of the group, or illustrative exemplars of the group.

[0145] Furthermore, the phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the items shown in the list may be used, and that only one of each item in the list may be required. In other words, "at least one of" means that any combination and any number of items from the list may be used, but not all of the items in the list are required. An item can be a particular object, thing, or category.

[0146] For example, "at least one of item A, item B, or item C" may include, but is not limited to, item A, item A and item B, or item B. An example of this may include item A, item B, and item C, or item B and item C. Of course, any combination of these items can be present. In some examples, "at least one of" can be, for example, but is not limited to, two items A, one item B, and ten items C, four items B and seven items C, or other suitable combinations.

[0147] Different instances of the term "embodiment" as used within this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. All data and data structures shown or described herein are merely examples, and in other embodiments, different amounts of data, types of data, fields, numbers and types of fields, field names, numbers and types of rows, records, entries, or organization of data may be used. In addition, any data may be combined with logic such that separate data structures may not be necessary. Thus, the preceding detailed description should not be taken in a limiting sense.

[0148] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many changes and modifications that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

[0149] While the present invention has been described with respect to specific embodiments, it is anticipated that alterations and modifications thereto will become apparent to those skilled in the art, and it is therefore intended that the following claims be interpreted to cover all such alterations and modifications as fall within the true spirit and scope of the invention.

[0150] An embodiment of the present disclosure provides a computer-implemented method for generating a performance report for a system adapted to execute a plurality of flows, the computer-implemented method including obtaining performance data associated with each of a plurality of running flows executed on the system; obtaining a plurality of templates, each template representing a function of a flow; linking each of the plurality of running flows to one of a plurality of templates representing a function of the linked running flow; generating, for each of the one or more templates, a performance summary based on the performance data associated with each of the plurality of running flows linked to the template; and generating a performance report based on the generated one or more performance summaries.

[0151] The method may further include, wherein generating the performance summary further includes: determining, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined amount; and, in response to determining, for each of the one or more templates, that the number of active flows linked to the template satisfies the condition, generating the performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0152] The method may further include, where obtaining the performance data includes obtaining, for each of the plurality of running flows, one or more of a latency of the running flow, a data rate per second of the running flow, a cost of the running flow, and an amount of time the running flow ran on the system.

[0153] The method may further include, wherein the performance summary includes a list of one or more of an average value of the performance data of the multiple running flows linked to the template, a range of the performance data of the multiple running flows linked to the template, and the performance data of the multiple running flows linked to the template.

[0154] The method may further include, wherein linking each of the plurality of active flows includes determining, for each of the plurality of active flows, a template of the plurality of templates that most closely represents a function of the active flow, and based on the determination, linking each of the plurality of active flows to one of the plurality of templates.

[0155] The method may further include each of the plurality of active flows being associated with one of the plurality of users, and each of the plurality of users being associated with one or more of the plurality of active flows. In some embodiments, the method may further include generating a performance report including receiving a performance report query from a user of the plurality of users, generating a performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on performance data associated with one or more of the plurality of active flows, and at least one of the one or more of the plurality of active flows being associated with the user, and transmitting the performance report to the user in response to receiving the performance report query.

[0156] The method may further include the system being a multi-tenant integrated cloud system, and each of the plurality of active flows being assigned a default processing capacity of the system.

[0157] Another embodiment of the present disclosure provides a computer-implemented method for updating a performance report for a system adapted to execute a plurality of flows, the computer-implemented method including: obtaining a performance report, the performance report based on one or more performance summaries associated with one or more of a plurality of templates; obtaining a plurality of test flows, each test flow associated with one of the plurality of templates; executing the set of the plurality of test flows on the system; obtaining test performance data associated with each of the test flows in the set of the plurality of test flows; generating, for at least one of the plurality of templates, a test performance summary based on the performance data associated with each of the test flows associated with the template; and updating the performance report based on the generated one or more test performance summaries.

[0158] The method may further include, where obtaining the test performance data includes obtaining, for each of the set of multiple test flows, one or more of a latency of the test flow, a data rate per second of the test flow, a cost of the test flow, and an amount of time the test flow executed on the system.

[0159] The method may further include executing the set of multiple test flows including determining the set of multiple test flows based on an idle processing capacity of the system, and executing the set of multiple test flows on the system.

[0160] The method may further include, wherein determining the set of the plurality of test flows further includes determining an idle processing capacity of the system, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, increasing an amount of the system processing capacity used by the test flows, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, decreasing an amount of the system processing capacity used by the test flows.

[0161] The method may further include, where increasing the amount of system processing capacity used by the test flows includes at least one of adding one or more test flows to the set of the plurality of test flows and increasing a data throughput of one or more of the test flows in the set of the plurality of test flows, and, where decreasing the amount of system processing capacity used by the test flows includes at least one of removing one or more test flows from the set of the plurality of test flows and decreasing a data throughput of one or more of the test flows in the set of the plurality of test flows.

[0162] The method may further include generating a performance report for the system adapted to execute the plurality of flows, further including updating the performance report, where executing the set of the plurality of test flows includes determining a set of the plurality of test flows based on an idle processing capacity of the system, and executing the set of the plurality of test flows on the system.

[0163] Another embodiment of the present disclosure provides an apparatus for generating a performance report for a system adapted to execute a plurality of flows, the apparatus comprising: an index unit configured to obtain performance data associated with each of a plurality of running flows; a mapping unit configured to obtain a plurality of templates, each template representing a function of a flow, and to link each of the plurality of running flows to one of the plurality of templates representing a function of the linked running flow; and a report generator configured to generate, for each of the one or more templates, a performance summary based on the performance data associated with each of the plurality of running flows linked to the template, and to generate a performance report based on the generated one or more performance summaries.

[0164] The apparatus may further include, wherein the report generator is further configured to: determine, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined amount; and, in response to determining, for each of the one or more templates, that the number of active flows linked to the template satisfies the condition, generate a performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0165] The apparatus may further include, wherein the index unit is configured to obtain, for each of the plurality of active flows, one or more of a latency of the active flow, a data rate per second of the active flow, a cost of the active flow, and an amount of time the active flow has run on the system.

[0166] The apparatus may further include, wherein the report generator is further configured to: receive a performance report query from the user; generate a performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on performance data associated with one or more of the plurality of running flows, where one of the one or more of the plurality of running flows is associated with the user; and, in response to receiving the performance report query, transmit the performance report to the user.

[0167] The apparatus may further include, wherein the system is a multi-tenant integrated cloud system, and each of the plurality of active flows is assigned a default throughput limit of the system.

[0168] Another embodiment of the present disclosure provides an apparatus for updating a performance report for a system adapted to execute a plurality of flows, the apparatus comprising: a report generator configured to obtain a performance report, the performance report being based on one or more performance summaries associated with one or more of a plurality of templates; a performance test driver configured to obtain a plurality of test flows, each test flow being associated with one of the plurality of templates, and to drive execution of the set of the plurality of test flows on the system; and an indexing unit configured to obtain test performance data associated with each of the plurality of sets of executed test flows, wherein the report generator is further configured to: generate, for at least one of the plurality of templates, a test performance summary based on the performance data associated with each of the test flows associated with the template; and update the performance report based on the generated one or more test performance summaries.

[0169] The apparatus may further include, wherein the index unit is configured to obtain, for each of the plurality of test flows, one or more of a latency of the test flow, a data rate per second of the test flow, a cost of the test flow, and an amount of time the test flow was executed on the system.

[0170] The apparatus may further include, wherein the performance test driver is configured to determine a set of a plurality of test flows based on an idle processing capacity of the system and drive execution of the set of the plurality of test flows on the system.

[0171] The apparatus may further include, wherein the performance test driver is further configured to determine an idle processing capacity of the system, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, increase an amount of the system processing capacity used by the test flow, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, decrease an amount of the system processing capacity used by the test flow.

[0172] The apparatus may further include, wherein the performance test driver is further configured to: add one or more test flows to the set of the plurality of test flows; increase a data throughput of one or more of the test flows in the set of the plurality of test flows; remove one or more test flows from the set of the plurality of test flows; and decrease a data throughput of one or more of the test flows in the set of the plurality of test flows.

[0173] An embodiment of the present disclosure also provides an apparatus for generating a performance report for a system configured to execute a plurality of flows, the apparatus comprising an indexing unit configured to obtain performance data associated with each of a plurality of active flows, the apparatus also comprising a mapping unit configured to obtain a plurality of templates, each template representing a function of a flow, and to link each of the plurality of active flows to one of the plurality of templates representing a function of the linked active flow, and a report generator configured to generate, for each of the one or more templates, a performance summary based on the performance data associated with each of the plurality of active flows linked to the template, and to generate a performance report based on the generated one or more performance summaries.

[0174] According to some embodiments, the report generator may be further configured to: determine, for each of the one or more templates, whether a number of active flows linked to the template satisfies a condition based on a predetermined quantity; and, in response to determining, for each of the one or more templates, that the number of active flows linked to the template satisfies the condition, generate a performance summary based on performance data associated with each of the plurality of active flows linked to the template.

[0175] According to some embodiments, the index unit may be configured to obtain, for each of the plurality of active flows, one or more of: a latency of the active flow, a data rate per second of the active flow, a cost of the active flow, and an amount of time the active flow has been running on the system.

[0176] According to some embodiments, the report generator may be further configured to: receive a performance report query from a user; generate a performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on performance data associated with one or more of the plurality of running flows, where one of the one or more of the plurality of running flows is associated with the user; and, in response to receiving the performance report query, transmit the performance report to the user.

[0177] According to some embodiments, the system may be a multi-tenant integrated cloud system, where each of a number of active flows is assigned a default throughput limit of the system.

[0178] According to some embodiments of the present disclosure, an apparatus is provided for updating a performance report for a system configured to execute a plurality of flows. The apparatus includes a report generator configured to obtain a performance report, the performance report based on one or more performance summaries associated with one or more of a plurality of templates. The apparatus also includes a performance test driver configured to obtain a plurality of test flows, each test flow associated with one of the plurality of templates, and to drive execution of a set of the plurality of test flows on the system, and an indexing unit configured to obtain test performance data associated with each of the set of the plurality of executed test flows. The report generator is further configured to perform, for at least one of the plurality of templates, generating a test performance summary based on the test performance data associated with each of the test flows associated with the template, and updating the performance report based on the generated one or more test performance summaries.

[0179] According to some embodiments, the index unit may be configured to obtain, for each of the plurality of test flows, one or more of a latency of the test flow, a data rate per second of the test flow, a cost of the test flow, and a time the test flow was executed on the system.

[0180] According to some embodiments, a performance test driver may be configured to determine a set of test flows based on an idle processing capacity of a system, and to drive execution of the set of test flows on the system.

[0181] In some embodiments, the performance test driver may be further configured to determine an idle processing capacity of the system, and in response to determining that the idle processing capacity of the system exceeds a predefined capacity threshold, increase an amount of the system processing capacity used by the test flow, and in response to determining that the idle processing capacity of the system falls below the predefined capacity threshold, decrease an amount of the system processing capacity used by the test flow.

[0182] According to some embodiments, the performance test driver may be further configured to add one or more test flows to the set of the plurality of test flows, increase a data throughput of one or more of the test flows in the set of the plurality of test flows, remove one or more test flows from the set of the plurality of test flows, and decrease a data throughput of one or more of the test flows in the set of the plurality of test flows.

Claims

1. 1. A computer-implemented method for generating a performance report for a system configured to execute a plurality of flows, comprising: obtaining performance data associated with each of a plurality of active flows on the system; obtaining a plurality of templates, each template representing a function of a flow; linking each of the plurality of active flows to one of the plurality of templates, the template representing a function of the linked active flow; generating, for each of one or more templates, a performance summary based on the performance data associated with each of the plurality of active flows linked to the template; and generating a performance report based on the generated one or more performance summaries.

2. generating said performance summary, determining, for each of the one or more templates, whether a number of active flows linked to the template satisfies a predefined volume-based condition; 2. The computer-implemented method of claim 1, further comprising: for each of the one or more templates, in response to determining that a number of the active flows linked to the template satisfy the condition, generating the performance summary based on the performance data associated with each of the plurality of active flows linked to the template.

3. Obtaining performance data For each of the plurality of active flows, the latency of said running flows; the data rate per second of said active flow; the cost of said in-service flow; and The computer-implemented method of claim 1 , further comprising obtaining one or more selected from the group consisting of: a time that the running flow was executed on the system.

4. The performance summary is an average value of the performance data for the plurality of active flows linked to the template; a range of the performance data for the plurality of active flows linked to the template; and The computer-implemented method of claim 1 , further comprising one or more selected from the group consisting of one or more lists of the performance data of the plurality of active flows linked to the template.

5. linking each of the plurality of active flows determining, for each of the plurality of active flows, the template of the plurality of templates that most closely represents the functionality of the active flow; and linking each of the plurality of active flows to one of the plurality of templates based on the determination.

6. The computer-implemented method of claim 1 , wherein each of the plurality of active flows is associated with one of a plurality of users, and each of the plurality of users is associated with one or more of the plurality of active flows.

7. generating the performance report, receiving a performance report query from a user of the plurality of users; generating the performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on the performance data associated with one or more of the plurality of running flows, at least one of the one or more of the plurality of running flows being associated with the user; The computer-implemented method of claim 6 , further comprising: in response to receiving the query for the performance report, transmitting the performance report to the user.

8. 2. The computer-implemented method of claim 1, wherein the system is a multi-tenant integrated cloud system, and each of the plurality of active flows is assigned a default processing capacity of the system.

9. 1. A computer-implemented method for updating a performance report for a system configured to execute a plurality of flows, comprising: obtaining a performance report, the performance report being based on one or more performance summaries associated with one or more of a plurality of templates; obtaining a plurality of test flows, each test flow being associated with one of the plurality of templates; executing the set of test flows on a system; obtaining test performance data associated with each of the test flows in the set of the plurality of test flows; generating, for at least one of the plurality of templates, a test performance summary based on test performance data associated with each of the test flows associated with the template; and updating the performance report based on the generated one or more test performance summaries.

10. Obtaining test performance data For each of the set of test flows, the latency of said test flow; the data rate per second of said test flow; the cost of said test flow; and 10. The computer-implemented method of claim 9, further comprising obtaining one or more selected from the group consisting of: a time that the test flow was executed on the system.

11. executing the set of the plurality of test flows; determining the set of test flows based on an idle processing capacity of the system; and executing the set of the plurality of test flows on the system.

12. determining the set of the plurality of test flows; determining the idle processing capacity of the system; increasing an amount of the idle processing capacity of the system used by the test flow in response to determining that the idle processing capacity exceeds a predefined capacity threshold; 12. The computer-implemented method of claim 11, further comprising: in response to determining that the idle processing power is below a predefined power threshold, reducing an amount of the idle processing power of the system used by the test flow.

13. Increasing the amount of the idle processing capacity of the system used by the test flow adding one or more test flows to the set of test flows; increasing a data throughput of one or more of the test flows in the set of the plurality of test flows; Reducing the amount of the idle processing power of the system used by the test flow removing one or more test flows from the set of test flows; and reducing the data throughput of one or more of the test flows in the set of the plurality of test flows.

14. A processor; and a computer-readable storage medium communicatively coupled to the processor and storing program instructions, the program instructions, when executed by the processor, causing the processor to: obtaining performance data associated with each of a plurality of active flows on the system; obtaining a plurality of templates, each template representing a function of a flow; linking each of the plurality of active flows to one of the plurality of templates, the template representing a function of the linked active flow; generating, for each of one or more templates, a performance summary based on the performance data associated with each of the plurality of active flows linked to the template; and generating a performance report based on the generated one or more performance summaries.

15. generating said performance summary, determining, for each of the one or more templates, whether a number of active flows linked to the template satisfies a predefined volume-based condition; and generating the performance summary based on the performance data associated with each of the active flows linked to the template in response to determining that, for each of the one or more templates, a number of the active flows linked to the template satisfy the condition.

16. Obtaining performance data For each of the plurality of active flows, the latency of said running flows; the data rate per second of said active flow; the cost of said in-service flow; and The system of claim 14 , further comprising obtaining one or more selected from the group consisting of: a time that the active flow has been executed on the system.

17. generating the performance report, receiving a performance report query from a user of a plurality of users; generating the performance report based on the generated one or more performance summaries, each of the one or more performance summaries based on the performance data associated with one or more of the plurality of running flows, at least one of the one or more of the plurality of running flows being associated with the user; 15. The system of claim 14, further comprising: in response to receiving the query for the performance report, transmitting the performance report to the user.

18. 15. The system of claim 14, wherein the system is a multi-tenant integrated cloud system, and each of the plurality of active flows is assigned a default processing capacity limit of the system.

19. A processor; and a computer-readable storage medium communicatively coupled to the processor and storing program instructions, the program instructions, when executed by the processor, causing the processor to: obtaining a performance report, the performance report being based on one or more performance summaries associated with one or more of a plurality of templates; obtaining a plurality of test flows, each test flow being associated with one of the plurality of templates; executing the set of test flows on a system; obtaining test performance data associated with each of the test flows in the set of the plurality of test flows; generating, for at least one of the plurality of templates, a test performance summary based on test performance data associated with each of the test flows associated with the template; and updating the performance report based on the generated one or more test performance summaries.

20. Obtaining test performance data For each of the set of test flows, the latency of said test flow; the data rate per second of said test flow; the cost of said test flow; and 20. The system of claim 19, further comprising obtaining one or more selected from the group consisting of: a time that the test flow was executed on the system.

21. executing the set of the plurality of test flows; determining the set of test flows based on an idle processing capacity of the system; and executing the set of the plurality of test flows on the system.

22. determining the set of the plurality of test flows; determining the idle processing capacity of the system; increasing an amount of the idle processing capacity of the system used by the test flow in response to determining that the idle processing capacity exceeds a predefined capacity threshold; 22. The system of claim 21, further comprising: in response to determining that the idle processing capacity is below a predefined capacity threshold, reducing an amount of the idle processing capacity of the system used by the test flow.

23. Increasing the amount of the idle processing capacity of the system used by the test flow adding one or more test flows to the set of test flows; increasing a data throughput of one or more of the test flows in the set of the plurality of test flows; Reducing the amount of the idle processing power of the system used by the test flow removing one or more test flows from the set of test flows; and reducing the data throughput of one or more of the test flows in the set of the plurality of test flows.

24. A computer program product causing a computer to carry out the method according to any one of claims 1 to 13.

Images