Generating and updating performance reports

Abstract

A computer-implemented method for generating and updating performance reports for systems adapted to run multiple streams is disclosed, which enables performance data for individual running streams to be compared to streams with similar functionality. This is achieved by linking each running stream to one of a plurality of templates that describe the functionality of the stream. Each template can have an associated performance summary generated for it based on performance data for running streams linked to the template. A performance report can be generated from the plurality of performance summaries. The performance report can then be updated by running a plurality of test streams on a spare capacity of the system while obtaining test performance data from the test streams. The test performance data can be compared to performance data for running streams linked to templates of the system.

Description

Background Technology

[0001] This disclosure relates to the performance of a reporting system, and more specifically, to generating performance reports for a system configured to run multiple streams.

[0002] Customers are increasingly accessing applications provided by third parties. These applications are hosted by third parties and typically delivered to customers via the Internet. This occurs in the form of Software as a Service (SaaS), Infrastructure as a Service (IaaS), and / or Platform as a Service (PaaS). For many applications, customers purchase multiple licenses or processing power to gain access to the application.

[0003] The system capacity required by a customer is typically defined by a capacity planning process. This process usually considers the type of processing the customer requires and then uses sample performance data to calculate the required capacity. However, capacity planning often involves sampling performance for a very limited set of scenarios, as these scenarios need to be manually configured for testing (or configured using limited automation). In reality, many aspects can vary, such as stream complexity, message size, and the number of backend calls required. Because of this variability, obtaining accurate capacity requirement forecasts is time-consuming and difficult. Therefore, forecasts of required capacity will often include 100% overhead to account for inaccurate forecasts. In this case, if the forecast is accurate, the customer may end up paying for twice the amount of capacity. Summary of the Invention

[0004] According to embodiments of this disclosure, a computer-implemented method is provided for generating performance reports for a system configured to run multiple streams. The method includes obtaining performance data associated with each of a plurality of running streams on the system. The method also includes obtaining a plurality of templates, each template describing the functionality of a stream. The method further includes linking each of the plurality of running streams to one of the templates, the template describing the functionality of the linked running stream. The method also includes generating a performance profile for each of one or more templates based on the performance data associated with each of the plurality of running streams linked to the template. The method further includes generating a performance report based on the generated one or more performance profiles.

[0005] A concept is proposed that links streams running on a system to templates describing the functionality of those streams. This enables the grouping of similar streams, and subsequently, the performance of the streams can be measured and compared. As a result, performance reports can be generated, where the performance of similar streams can be seen and compared. Thus, the proposed embodiment can provide context for performance data that would otherwise be unavailable, such as the average performance or possible range of similar streams. Such contextual information is useful to customers when evaluating performance and / or for informing them of changes to be made to the streams or the system on which they run.

[0006] By grouping the flows running on the system into templates, multiple similar sets of flows can be defined. Performance data for all flows can then be collected and compared. This allows performance to be compared to other flows with similar functionalities. By generating performance reports from the performance data, customers can easily see how their flows compare to other similar flows, and this information can inform the customer's capacity planning process.

[0007] Therefore, some proposed embodiments may involve generating a performance report containing contextualized performance data for each stream running on the system. Streams can be linked to templates that describe the functionality of the streams. Many streams can be linked to each template, and when performance data for each stream is obtained, the performance data of streams with similar functionality can be compared. A performance summary can be generated for each template, which may include, for example, the average performance, best performance, and a list of all streams and their associated performance for each stream linked to that template.

[0008] Therefore, some of the proposed embodiments can provide the benefit of providing performance data that accurately reflects the actual performance characteristics of the system. Specifically, accurate and contextualized performance data can be provided, which can greatly improve the capacity planning process and help avoid various performance-related problems (e.g., caused by insufficient capacity provision). Such information is particularly useful for new customers planning to use the system, for example, by providing an understanding of how much capacity they will need. As a further example, a customer can run a stream without generating a performance number for the stream. Therefore, users may prefer to match streams with templates and corresponding performance numbers from other customers.

[0009] Furthermore, performance reports generated using data and templates from numerous customers should preferably avoid / minimize information leakage from any given customer. This information could, for example, involve flow design, details about message rates, or other business-sensitive information. To address this issue, embodiments can be configured to use only templates with the corresponding performance figures in the report if the data used to generate the report comes from more than one customer (or potentially more than a defined number of customers). In this way, the ability to infer usage from individual customers from performance reports can be reduced or prevented.

[0010] According to some embodiments, generating a performance profile may further include: determining, for each of one or more templates, whether the number of runtimes linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtimes linked to the template satisfies the condition, generating a performance profile for each of the one or more templates based on performance data associated with each of the plurality of runtimes linked to the template.

[0011] Ensuring that each template has a runtime flow with more than a certain number of links ensures that a performance profile is generated only if there is sufficient data to provide meaningful comparisons. This allows outlier data to have a reduced impact.

[0012] According to some embodiments, obtaining performance data may include, for each of the plurality of running flows, obtaining one or more selected from the group consisting of: the latency of the running flow; the data rate per second of the running flow; the cost of the running flow; and the time taken to execute the running flow on the system. Thus, latency, data rate, and cost can be used to provide a measurement of the performance of each running flow. By recording the time the flows run on the system, a performance profile can compare running flows occurring at a specific time of day and provide information about bottlenecks to further inform the capacity planning process. It should be understood that other performance metrics may be used.

[0013] According to some embodiments, a performance profile may include one or more selected from the group consisting of: the average value of the performance data of the plurality of run streams linked to the template; the range of the performance data of the plurality of run streams linked to the template; and a list of one or more performance data of the plurality of run streams linked to the template.

[0014] By processing data from all running streams linked to the template, further context can be provided regarding the performance of each running stream. For example, if the performance of a given stream is significantly lower than the average performance of all running streams linked to the template, this could indicate a need for greater capacity.

[0015] According to some embodiments, linking each of a plurality of runtime flows may include: for each of the plurality of runtime flows, determining the template among the plurality of templates that most closely describes the functionality of the runtime flow; and based on the determination, linking each of the plurality of runtime flows to one of the plurality of templates.

[0016] By ensuring that execution flows are linked to templates that best describe the functionality of the flows, data from each execution flow can be compared with other similar execution flows. This provides meaningful context.

[0017] According to some embodiments, each of the multiple run streams may be associated with one of the multiple users, and each of the multiple users may be associated with one or more of the multiple run streams.

[0018] Each run can be associated with multiple users or clients, and the users for each run can be different. This allows the performance of a user's run traffic to be compared with similar run traffic from various other users, each of whom can access different capabilities of the system.

[0019] According to some embodiments, generating a performance report may further include: receiving a performance report query from a user among the plurality of users; generating the performance report based on one or more generated performance profiles, each of the one or more performance profiles being based on performance data associated with one or more of the plurality of running streams, at least one of the one or more running streams being associated with the user; and sending the performance report to the user in response to receiving the performance report query.

[0020] Users can request performance reports, and these reports can be generated to display the performance of the running streams associated with those users. Performance summaries that are not linked to running streams associated with a user may be excluded because they are not relevant to that user.

[0021] According to some embodiments, the system may be a multi-tenant integrated cloud system, and each of the multiple running streams may be allocated a predetermined processing capacity of the system.

[0022] A certain amount of system processing power can be allocated to each user running on the system to run their associated runtime flows. If less processing power is allocated to a user, the runtime flow associated with that user may perform worse than another user running a similar runtime flow with more processing power. This provides for the performance disparity of runtime flows in a cloud-based multi-tenant system.

[0023] According to some embodiments of this disclosure, a computer-implemented method is provided for updating a performance report of a system configured to run multiple streams. The method includes obtaining a performance report based on one or more performance profiles associated with one or more of a plurality of templates. The method also includes obtaining a plurality of test streams, wherein each test stream is associated with one of the plurality of templates. The method further includes executing a set of the plurality of test streams on the system. The method also includes obtaining test performance data associated with each of the plurality of executed test streams. The method further includes generating a test performance profile for at least one of the plurality of templates based on the test performance data associated with each of the test streams associated with the templates. The performance report is updated based on the generated one or more test performance profiles.

[0024] Another aspect of embodiments of this disclosure is the generation of additional performance data to provide further context to the performance data of streams linked to the template. This is achieved by running a test stream on the system, which is based on the template and is therefore linked to streams running on the system linked to the template. In this way, test performance data can be generated, for example, reflecting the performance of the streams running with access to an ideal level of capability, and thus practically achieving the optimal performance of the streams. This data can then be provided to the user to further inform the capability planning process (e.g., by providing information about the performance that one or more streams can achieve).

[0025] The proposal suggests further utilizing templates linked to multiple flows to obtain multiple test flows. These test flows can then be run on the system to provide more performance data in the form of test performance data, and thus provide more context to the multiple flows linked to the templates on which the test flows are based. This can be used, for example, to update performance reports and thus provide users / customers with more information to inform their capability planning process.

[0026] According to some embodiments, obtaining test performance data may include: for each of the plurality of test flows in the set, obtaining one or more selected from the group consisting of: the latency of the test flow; the data rate per second of the test flow; the cost of the test flow; and the time taken to execute the test flow on the system.

[0027] According to some embodiments, executing the set of multiple test flows may include: determining the set of multiple test flows based on the idle processing capacity of the system; and executing the set of multiple test flows on the system.

[0028] This leads to the selection of test flows taking into account the system's backup processing capacity. This means, for example, that the system's backup capacity utilized by a test flow can be maximized without affecting the portion of the system used by other flows. For instance, test flows to be executed can be prioritized based on which template has recently had test flows executed on the system based on it, or by the number of flows linked to the template on which the test flow is based.

[0029] According to some embodiments, determining the set of the plurality of test flows may further include: determining the idle processing capacity of the system; increasing the amount of processing capacity of the system used by the test flows in response to determining that the idle processing capacity of the system is higher than a predetermined capacity threshold; and decreasing the amount of processing capacity of the system used by the test flows in response to determining that the idle processing capacity of the system is lower than a predetermined capacity threshold.

[0030] In this way, the test flow can ramp up or down based on how much system processing power the actual running flow is using. Reserve capacity overhead can be maintained to ensure the running flow has sufficient processing power.

[0031] According to some embodiments, increasing the processing power of the system used by a test stream may include at least one selected from the group consisting of: adding one or more test streams to a set of multiple test streams; and increasing the data throughput of one or more of the test streams in the set of multiple test streams. According to some embodiments, decreasing the processing power of the system used by a test stream may include at least one of: removing one or more test streams from the set of multiple test streams; and decreasing the data throughput of one or more of the test streams in the set of multiple test streams.

[0032] Adding test streams allows you to obtain performance data for comparison with more running streams. Increasing data throughput allows you to measure test stream performance data for comparison with running streams that have high data throughput. You can remove test streams and reduce data throughput to ensure sufficient processing power overhead.

[0033] Embodiments of this disclosure also provide a method for generating a performance report for a system configured to run multiple streams, according to some embodiments, and further include updating the performance report according to other embodiments.

[0034] Embodiments of this disclosure also provide a system including a processor and a computer-readable storage medium communicatively coupled to the processor and storing program instructions that, when executed by the processor, cause the processor to perform a method. The method executed by the processor includes obtaining performance data associated with each of a plurality of running flows on the system. The method also includes obtaining a plurality of templates, each template describing the functionality of a flow. The method further includes linking each of the plurality of running flows to one of the plurality of templates, the template describing the functionality of the linked running flow. The method also includes generating a performance profile for each of one or more templates based on the performance data associated with each of the plurality of running flows linked to the template. The method further includes generating a performance report based on the generated one or more performance profiles.

[0035] According to some embodiments, generating a performance profile may further include: determining, by the processor and for each of the one or more templates, whether the number of runtime streams linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtime streams linked to the template satisfies the condition, generating a performance profile for each of the one or more templates based on performance data associated with each of the plurality of runtime streams linked to the template.

[0036] According to some embodiments, obtaining performance data may include: by the processor and for each of the plurality of running streams, obtaining one or more selected from the group consisting of: the latency of the running stream; the data rate per second of the running stream; the cost of the running stream; and the time to execute the running stream on the system.

[0037] According to some embodiments, a performance profile may include one or more selected from the group consisting of: the average value of the performance data of the plurality of run streams linked to the template; the range of the performance data of the plurality of run streams linked to the template; and a list of one or more performance data of the plurality of run streams linked to the template.

[0038] According to some embodiments, generating a performance report may further include: the processor receiving a performance report query from a user among the plurality of users; generating the performance report based on one or more generated performance profiles, each of the one or more performance profiles being based on performance data associated with one or more of the plurality of running streams, at least one of the one or more running streams being associated with the user; and sending the performance report to the user in response to receiving the performance report query.

[0039] According to some embodiments, the system may be a multi-tenant integrated cloud system, and each of the multiple running streams may be allocated a predetermined processing capacity of the system.

[0040] According to some embodiments of this disclosure, a system is provided including a processor and a computer-readable storage medium communicatively coupled to the processor and storing program instructions that, when executed by the processor, cause the processor to perform a method. The method executed by the processor includes obtaining a performance report based on one or more performance profiles associated with one or more of a plurality of templates. The method also includes obtaining a plurality of test streams, wherein each test stream is associated with one of the plurality of templates. The method further includes executing a set of the plurality of test streams on the system. The method also includes obtaining test performance data associated with each of the plurality of executed test streams. The method further includes generating a test performance profile for at least one of the plurality of templates based on the test performance data associated with each of the test streams associated with the templates. The performance report is updated based on the generated one or more test performance profiles.

[0041] According to some embodiments, obtaining test performance data may include: the processor obtaining one or more of the following selected from the group consisting of: the latency of the test stream; the data rate per second of the test stream; the cost of the test stream; and the time to execute the test stream on the system for each of the plurality of test streams.

[0042] According to some embodiments, executing the set of multiple test flows may include: the processor determining the set of multiple test flows based on the idle processing capacity of the system; and executing the set of multiple test flows on the system.

[0043] According to some embodiments, determining the set of the plurality of test streams may further include: the processor determining the idle processing capacity of the system; in response to determining that the idle processing capacity system is higher than a predetermined capacity threshold, increasing the amount of processing capacity of the system used by the test streams; and in response to determining that the idle processing capacity system is lower than a predetermined capacity threshold, decreasing the amount of processing capacity of the system used by the test streams.

[0044] According to some embodiments, increasing the processing power of the system used by a test stream may include at least one selected from the group consisting of: adding one or more test streams to a set of multiple test streams; and increasing the data throughput of one or more of the test streams in the set of multiple test streams. According to some embodiments, decreasing the processing power of the system used by a test stream may include at least one of: removing one or more test streams from the set of multiple test streams; and decreasing the data throughput of one or more of the test streams in the set of multiple test streams.

[0045] According to some embodiments of this disclosure, a computer program product is provided for generating a performance report for a system configured to run multiple streams. The computer program product includes a computer-readable storage medium having program instructions embodied therein, which are executable by a processor to cause the processor to perform a method. The method includes obtaining performance data associated with each of a plurality of running streams executing on the system. The method also includes obtaining a plurality of templates, each template describing the functionality of a stream. The method further includes linking each of the plurality of running streams to one of the plurality of templates, the template describing the functionality of the linked running stream. The method also includes generating a performance profile for each of one or more templates based on the performance data associated with each of the plurality of running streams linked to the template. The method further includes generating a performance report based on the generated one or more performance profiles.

[0046] The foregoing description is not intended to depict every illustrated embodiment or every implementation of this disclosure. These and other aspects of this disclosure will be apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0047] To better understand this disclosure, and to more clearly illustrate how it is implemented, reference will now be made to the accompanying drawings by way of example only, wherein:

[0048] Figure 1 A cloud computing environment according to an embodiment of the present invention is described;

[0049] Figure 2 An abstract model layer according to an embodiment of the present invention is described;

[0050] Figure 3A A block diagram depicting three templates and a set of example streams associated with each template is shown according to embodiments of the present disclosure;

[0051] Figure 3B A block diagram depicting three users and an example set of streams associated with each user is shown according to an embodiment of the present disclosure;

[0052] Figure 4 A flowchart of a method for generating a performance report according to an embodiment of the present disclosure is shown;

[0053] Figure 5 A flowchart of a method for updating a performance report according to an embodiment of the present disclosure is shown;

[0054] Figure 6 A block diagram of an apparatus configured to generate and update performance reports according to embodiments of the present disclosure is shown; and

[0055] Figure 7 A high-level block diagram of an example computer system, according to embodiments of the present disclosure, is shown that can be used to implement one or more of the methods, tools, and modules described herein, as well as any related functions. Detailed Implementation

[0056] This disclosure will be described with reference to the accompanying drawings.

[0057] It should be understood that the accompanying drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used in all the drawings to denote the same or similar parts.

[0058] In the context of this application, where the embodiments of this disclosure constitute a method, it should be understood that such a method can be a process for execution by a computer, i.e., it can be a computer-implementable method. Therefore, the various steps of the method can reflect various parts of a computer program, such as various parts of one or more algorithms.

[0059] Application and service integration typically involves: (i) configuring a flow (also known as an "application flow" or "integration flow") that describes the data flow between applications and the processing performed on the data inputs and outputs; and then (ii) parsing said data and transforming it into a format that can be used by another application or service. Such flows are typically stored in formats such as Extensible Markup Language (XML), JavaScript Object Notation (JSON), or YAML, and can be interpreted by a specific runtime as a set of instructions for interacting with other systems (where each discrete set of instructions is called a node). A flow is configured by concatenating sequences of nodes, where each node is customized to perform a specific task (e.g., transformation, filtering, aggregation, etc.). The integration product of the on-premises or off-premises platform then processes the data according to the flow's specifications. On-premises platforms are well-established and considered to offer a high level of security because data is stored and processed internally, such as within a private internal network. Off-premises platforms (such as cloud computing resources) are a relatively new and developing concept. Generally, references to off-site resources or platforms are considered to refer to the concept of enabling ubiquitous, convenient, and on-demand access via the Internet to a shared pool of configurable off-site (e.g., remotely located) computing resources, such as networks, applications, servers, storage devices, and functionalities accessible via the Internet. Conversely, references to on-site resources or platforms are considered to refer to the concept of local or dedicated computing resources, such as networks, servers, storage devices, and applications, located within / behind a local or virtual boundary (typically behind a firewall).

[0060] According to the proposed embodiments, a method and / or apparatus are provided for generating performance reports for a system configured to run multiple streams. This proposed method / apparatus obtains performance data associated with each of a plurality of running streams executed on the system. A plurality of templates are also obtained (each template describing the functionality of a stream). Each running stream is then linked to a template describing the functionality of the linked running stream. A performance summary for that template is then generated based on the performance data associated with each of the plurality of running streams linked to each of one or more templates. In other words, for each template, a performance summary is generated based on the performance of the determined streams linked to the template. Such a summary can represent the performance associated with a specific function. A performance report can then be created based on the performance summary.

[0061] Links between flows running on the system and templates describing the functionality of those flows enable grouping of flows that conform to a given template (i.e., those related to the functionality of the template). Performance data reflecting the performance of each flow running on the system can be obtained. A performance profile can then be generated, where the performance data of flows conforming to a given template can be compared and contrasted, allowing the performance data collected for each flow to be contextualized. This context is useful to customers when evaluating performance and for informing them of changes that will be made to the flow or the system on which the flow runs.

[0062] It is suggested that during the capacity planning process, any missing customer information could relate to the performance of the customer's flow and how that performance compares to increased capacity. By linking flows running on the system to templates, multiple similar sets of flows can be obtained. Performance data for all these flows can then be collected and compared. This allows performance to be compared to other flows with similar functionality, but with access to different system processing capacities, for example. By subsequently generating performance reports, customers can easily see how their flow's performance compares to other similar flows, and this informs the customer's capacity planning process.

[0063] In some embodiments, generating a performance profile further includes: determining, for each of one or more templates, whether the number of runtimes linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtimes linked to the template satisfies the condition, generating a performance profile for each of the one or more templates based on performance data associated with each of the plurality of runtimes linked to the template.

[0064] For example, this embodiment ensures that a performance profile is generated only when there are a sufficient number of running streams linked to it, to ensure meaningful comparisons and to ensure that outlier data has a reduced impact. The result is a performance profile that presents data reflecting the performance of a wide range of similar streams running under different conditions.

[0065] According to some embodiments, obtaining performance data includes: for each of the plurality of running streams, obtaining one or more of the following: the latency of the running stream; the data rate per second of the running stream; the cost of the running stream; and the time to execute the running stream on the system.

[0066] Templates linked to multiple streams can be further utilized to obtain multiple test streams. These test streams can then be run on the system to provide more performance data in the form of test performance data, and thus provide more context to the multiple streams linked to the templates on which the test streams are based. This can be used to update performance reports, and thus give users / customers more information to inform their capability planning process.

[0067] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0068] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0069] The characteristics are as follows:

[0070] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0071] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0072] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0073] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0074] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0075] The service model is as follows:

[0076] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0077] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0078] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0079] The deployment model is as follows:

[0080] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist inside or outside a building.

[0081] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0082] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0083] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0084] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0085] Now for reference Figure 1The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate, such as, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 1 The types of computing devices 54A-N shown are for illustrative purposes only, and computing node 10 and cloud computing environment 50 can communicate with any type of computing device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0086] Now for reference Figure 2 This demonstrates a cloud computing environment of 50 ( Figure 1 This provides a set of functional abstractions. It should be understood beforehand that... Figure 2 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0087] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; storage 65; and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0088] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0089] In one example, management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking when utilizing resources in the cloud computing environment, as well as billing or invoicing for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud resource allocation and management to meet the required service level. Performance reporting 85 is used to generate performance reports based on the proposed concepts as detailed herein.

[0090] Workload 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: drawing and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis and processing 94; transaction processing 95; and mobile desktop 96.

[0091] The illustrative embodiments can be used in many different types of distributed processing environments. To provide context for describing the elements and functions of the illustrative embodiments, the accompanying drawings are provided as example environments in which aspects of the illustrative embodiments can be implemented. It should be understood that the drawings are merely exemplary and are not intended to assert or imply any limitation regarding the environment in which aspects or embodiments of the invention can be implemented. Many modifications can be made to the described environment without departing from the spirit and scope of the invention. Furthermore, apparatus or systems embodying one or more of the proposed concepts can take the form of any of a variety of different processing devices, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, off-site and on-site devices may include portable computing devices configured with flash memory to provide non-volatile memory for storing, for example, operating system files and / or user-generated data. Therefore, the system can essentially be any known or later-developed processing system without architectural limitations.

[0092] The proposed concept can enhance a system by providing an apparatus or method for generating performance reports for a system running multiple streams. Such a proposal can extend or improve capacity planning and / or capacity allocation.

[0093] To help understand the concepts presented, reference will now be made. Figure 3A and 3B Describe an exemplary embodiment.

[0094] Figure 3AA block diagram 310 is shown depicting three templates and a set of example streams 318 associated with each template, according to an embodiment of the present disclosure.

[0095] A stream can be thought of as a set of actions connected together to accomplish a task. For example, a stream can be an integration, where input to one system or application is connected to a call to another system. A running stream can be a stream that is running or has been running on a system configured to run multiple streams.

[0096] A template describes a pattern for a stream, or it can describe patterns for more than one stream. Streams can be created from templates. When a template describes a pattern for more than one stream, there can be mutable elements within the patterns described by the template, allowing more than one stream to be created from the template. For example, a set of templates can be generated from a set of streams, such that 100 streams generate 100 templates, 100 streams generate 50 templates, or 100 streams generate 10 templates.

[0097] like Figure 3A As shown, template #1 312 best describes the pattern or function of streams #2, #5, #6, and #8. Template #2 314 best describes the pattern or function of streams #1, #3, and #7. Template #3 316 best describes the pattern or function of stream #4. If there exists a stream whose pattern or function cannot be described by any existing template, a new template can be generated from said stream. If there exists a stream whose pattern or function cannot be described by any existing template, a new template can be generated from said stream and one or more other streams. If there exists a stream whose pattern or function cannot be described by any existing template, a new template can be generated by generalizing from existing templates.

[0098] A flow can be linked to a template that describes the function or pattern of the flow. A flow can be linked to a template that best describes the function or pattern of the flow. The template that best describes the function or pattern of the flow can be determined by measuring the variability in the pattern of the flow described by the template and the actual pattern of the flow.

[0099] Figure 3B A block diagram 320 is shown depicting three users and an example stream set 328 associated with each user, according to an embodiment of the present disclosure.

[0100] This system can run multiple execution streams. Each of these streams can originate from a user associated with it. Each user can have multiple execution streams. For example... Figure 3BAs shown, user #1 322 is associated with run flows #5 and #6. User #2 324 is associated with run flows #1, #2, #3, and #4. User #3 326 is associated with run flows #7 and #8. Run flows associated with a user can have similarity to run flows associated with another user, such that the flows are linked to the same template.

[0101] Figure 4 A flowchart 400 illustrating a method for generating a performance report according to embodiments of the present disclosure is shown.

[0102] In step 402, performance data from multiple running streams is obtained. This may include, but is not limited to, measuring the latency of the running streams, measuring the data rate through the running streams, measuring the system cost in terms of memory for the running streams, and measuring the system cost in terms of processing for the running streams. The runtime of the running streams on the system may also be recorded.

[0103] In step 404, multiple templates are obtained. These templates can be obtained from a template database. Alternatively, templates can be generated from multiple runtime flows. Templates can describe the functionality of a flow, and they can describe the functionality of one or more of the multiple runtime flows.

[0104] In step 406, each of the multiple running flows is linked to a template describing the functionality of the running flow. The template linked to the running flow can describe the running flow. The best fit can be determined by minimizing the variability between the pattern of the flow described by the template and the pattern of the running flow. Alternatively, a running flow can be linked to a template describing the functionality of the fewest number of flows, while still describing (or most closely describing) the functionality of the running flow.

[0105] A run stream can be linked to only one of several templates. This provides the benefit of linking only to the template that best describes the functionality of the run stream. Alternatively, a run stream can be linked to more than one of several templates. This provides the benefit of comparing with a wider range of different streams, resulting in more performance data points for comparison.

[0106] In step 408, a performance summary is generated for each of the multiple templates. If the number of run streams linked to a template is less than a certain number, a performance summary may not be generated for that template. For example, if only one run stream is linked to the template, no performance summary may be generated. This ensures that each generated performance summary has a sufficient number of run streams' performance data so that outliers are compensated for.

[0107] Generating a performance profile for a given template may include, but is not limited to, calculating the average performance data of all running streams linked to the template, calculating the range of performance data of all running streams linked to the template, compiling a list of performance data of all running streams linked to the template, and / or calculating the maximum and minimum performance data of all running streams linked to the template.

[0108] Performance profiles can also be generated using performance data from running streams at a given time, making them specific to data from streams running on the system during the morning, afternoon, a specific hour of the day, or a specific day of the week. Performance data from streams running at different times of the day can also be compared and contrasted in the performance profiles to help users identify times of bottlenecks or excess capacity. This will aid in the capacity planning process, as well as the planning process regarding when to run certain streams on the system.

[0109] In step 410, a performance report is generated. This may be based on one or more performance summaries generated in step 408.

[0110] In some embodiments, a performance report query is received from a user. The user can be associated with multiple run flows on the system that are linked to multiple templates. Upon receiving the performance report query, a performance report is generated. In this case, the performance report can be generated based on one or more performance summaries, each of which is associated with a template linked to one or more run flows associated with the user. This means that only the performance summary relevant to the user is presented to the user in the performance report. The performance report is then sent to the user when it is generated.

[0111] In one example, the user can select which performance data to include in the performance report and how the performance data is processed to generate each performance summary on which the performance report is based. In another example, the performance data and the way it is used to generate each performance summary on which the performance report is based are determined automatically.

[0112] For example, a user can associate with two running streams. Each running stream can be linked to its own template. Performance data for each running stream is obtained and then compiled into a performance summary, including performance data from other running streams linked to the template. The performance summary is then used to generate a performance report. The performance report is then delivered to the user. The performance report may include performance data for the two running streams associated with the user, the average performance data of other streams similar to the two running streams, and the best performance data of other streams similar to the two running streams. Additionally, the performance data for each running stream at different times can be presented in the performance report.

[0113] In other words, performance report generation can be described as follows: Each user's execution flow is matched with a template that best describes the functionality of that flow. In practice, all execution flows are categorized into groups that conform to a given template. Performance data collected from the execution flows is then stored in the form of performance summaries against their corresponding templates. This is only possible if a certain number of different execution flows are linked to the same template to anonymize the performance data. The performance summaries can then be collected and added to an automatically generated performance report that shows the average performance metric for each template.

[0114] Figure 5 A flowchart 500 illustrating a method for updating a performance report according to an embodiment of the present disclosure is shown.

[0115] In step 502, performance reports relating to the performance of multiple running streams are obtained. The performance reports may be based on one or more performance profiles, each of which may be associated with a template. Each template may be linked to multiple running streams.

[0116] In step 504, multiple test flows are obtained, each associated with a template. For each template associated with each performance summary in the performance report, at least one test flow may exist. Each test flow can be generated from the template. Alternatively, each test flow can be obtained from a database of test flows. Each test flow may share common functionality with one or more of the multiple run flows.

[0117] In step 506, a set of multiple test streams are executed, and test performance data for each test stream is obtained.

[0118] The set of multiple test flows is determined based on the system's backup capacity. This capacity could be processing power or storage capacity. The amount of capacity used by each test flow is determined by the system's backup capacity.

[0119] If the system's backup capacity exceeds a certain amount or percentage threshold, the number of test streams on the system can be increased. Alternatively, if the system's backup capacity exceeds a certain amount or percentage threshold, the data throughput or drive rate of test streams already running on the system can be increased.

[0120] If the system's backup capacity falls below a certain amount or percentage threshold, the number of test streams on the system can be reduced. Alternatively, if the system's backup capacity falls below a certain amount or percentage threshold, the data throughput or drive rate of test streams already running on the system can be reduced.

[0121] In this way, the system's available capacity for running flows remains at an acceptable level to accommodate changes in the running flow load. The system's available capacity can be continuously monitored, and the set of test flows can be re-evaluated on a continuous basis to ensure the system's acceptable availability.

[0122] For example, multiple test flows can be selected from a set of test flows to be run on the system based on the last time a test flow was run, the last time a test flow associated with a given template was run, and / or the number of run flows that have similar patterns or functions. In this way, a variety of test flows can be run, thereby obtaining test performance data related to a large number of run flows.

[0123] As each test stream runs on the system, test performance data for each test stream is obtained. This may include, but is not limited to, measuring the latency of the test stream, measuring the data rate through the test stream, measuring the system cost in terms of memory for the test stream, and measuring the system cost in terms of processing for the test stream. The runtime of the test stream on the system may also be recorded.

[0124] In step 508, a test performance summary is generated. The test performance summary is generated based on the test performance data obtained in previous steps. Each test performance summary involves one of several templates associated with the performance report.

[0125] Generating a test performance profile for a given template may include, but is not limited to, calculating the average of the test performance data for all test flows linked to the template, calculating the range of the test performance data for all test flows linked to the template, compiling a list of the test performance data for all test flows linked to the template, and / or calculating the maximum and minimum values ​​of the test performance data for all test flows linked to the template.

[0126] Test performance profiles can also be generated using test performance data from test streams at a given time, making the profiles specific to the test performance data from individual streams that ran on the system in the morning, afternoon, during a specific hour of a specific day or week, or throughout the day. Test performance data from test streams running at different times of the day can also be compared and contrasted in the test performance profiles to help users identify bottlenecks or periods of excess capacity. This will aid in the capacity planning process, as well as the planning of when to run batches of streams on the system or when to run streams to exceed a certain level of performance.

[0127] In step 510, the performance report is updated. A test performance summary is added to the performance report. A test performance summary associated with a template is added, which is linked to the performance summary in the performance report. In this way, test performance data related to the running stream included in the performance report is added. This provides the user with indications about the cost, latency, and potential maximum throughput of their running stream, as the user can compare the performance of their running stream to the performance of other running streams on the system from the original performance report, and also to the performance of the test stream.

[0128] In other words, performance report updates can be described as follows. For each identified template, a test stream can be manually created in the same way as a stream is typically created from a template. In practice, the tests running on the system can be based on a subset of the templates. This subset can be determined based on how many run streams are associated with each template and how different the templates are from other templates. This subset of test streams can then be run on the system in consecutive cycles, with the drive rate increasing until it reaches a percentage of the overall system processing capacity. Once a threshold is exceeded, the test stream will "ramp back" to keep the available processing power for the run streams at an acceptable level. The test performance data from the test streams can then be combined with the number of run stream performance data points in the performance report. This will allow the report to show the rate achieved by an actual user running the stream against a given template, and to show the maximum possible rate achieved by the test streams. Thus, users can understand the cost, latency, and potential maximum throughput. It can be expected that the number of data points from the test streams will be significantly fewer than the number of performance data points from the run streams.

[0129] Performance reports can be updated periodically in the manner described above. For example, performance reports can be updated whenever a performance report is generated, whenever a user submits a performance report query, or whenever a performance summary is generated.

[0130] Figure 6 A block diagram 600 depicting an apparatus for generating and updating performance reports according to an embodiment of the present disclosure is shown.

[0131] System 610 is configured to run multiple streams, including multiple running streams 612 and multiple test streams 614. System 610 can be a cloud-based multi-tenant system configured to run multiple streams from multiple users, each with access to specific processing capabilities of the system. System 610 can be a cloud-based integrated product, such as a cloud-based application (examples of which are known and widely available).

[0132] Mapping unit 620 can be configured to store templates in template storage 622. Mapping unit 620 can also be configured to store links between run streams and test streams, as well as templates, in template stream link storage 624. Mapping unit 620 is used to obtain multiple templates and link each run stream to one or more of said templates. Mapping unit 620 can be configured to determine templates that best describe the functionality and / or patterns of the run streams.

[0133] The metric unit 630 is configured to acquire performance data associated with each of the plurality of run streams 612 on the system 610. The metric unit 630 may also be configured to store the performance data associated with each of the plurality of run streams on the system 612 in the performance data storage 632.

[0134] The metric unit 630 is also configured to obtain test performance data associated with each of the plurality of test streams 614 on the system 610. The metric unit 630 may be further configured to store the test performance data associated with each of the plurality of test streams 614 on the system 610 in a test performance data storage 634.

[0135] The metric unit 630 can be configured to obtain the latency, data rate, operating cost, and runtime of the stream running on the system 610.

[0136] Performance test driver 640 is configured to acquire multiple test streams, determine a set of test streams that should run on the system based on the spare processing capacity of system 610, and drive the set of multiple test streams 614 on system 610. Performance test driver 640 may be configured to store the test streams in test stream storage 642. Performance test driver 640 may also be configured to determine the idle capacity of system 610 and increase or decrease the capacity of system 610 used by test streams 614. Performance driver 640 may be further configured to add or remove test streams from the set of test streams 614 on system 610, and increase or decrease the data throughput of the set of multiple test streams 614 on system 610.

[0137] Report generator 650 is configured to generate a performance summary and store the performance summary in performance summary storage 652. For each template, report generator 650 can be configured to obtain from mapping unit 620 which runtime flows are associated with the template, and then compile the performance data associated with the runtime flows from measurement unit 630 into a performance summary. Compiling the performance data may include averaging the performance data, determining the range of the performance data, and / or determining the maximum and minimum values ​​of the performance data. If a template has fewer than a certain number of runtime flows linked to it, performance report generator 650 may not generate a performance summary for that template.

[0138] The report generator 650 is also configured to generate a test performance summary and store the summary in a test performance summary storage 654. The report generator 650 can be configured to obtain from the mapping unit 620 which test flows are associated with the template, and then compile the test performance data associated with the test flows from the measurement unit 632 into the test performance summary. Compiling the test performance data may include averaging the test performance data, determining the range of the test performance data, and / or determining the maximum and minimum values ​​of the test performance data.

[0139] The report generator 650 is also configured to generate and update performance reports. The report generator 650 is configured to store the performance reports in the performance report storage 656.

[0140] Report generator 650 can generate performance reports based on multiple performance profiles. Report generator 650 can be configured to receive performance report queries from a user, generate performance reports based on performance profiles linked to the user via a runtime flow linked to templates associated with the performance profiles, and send the performance reports to the user. Alternatively, performance report generator 650 can also be configured to update the performance reports using test performance profiles before sending the performance reports to the user. Report generator 650 can be configured to store performance reports and continuously update them. Alternatively, report generator 650 can be configured to store performance reports and update them when a performance report query is received from a user.

[0141] Now for reference Figure 7 This document illustrates a high-level block diagram of an example computer system 1101 that can be used to implement one or more of the methods, tools, and modules described herein, as well as any associated functionality (e.g., using one or more processor circuits or a computer processor) according to embodiments of the present disclosure. In some embodiments, the main components of the computer system 1101 may include one or more CPUs 1102, a memory subsystem 1104, a terminal interface 1112, a storage interface 1116, an I / O (input / output) device interface 1114, and a network interface 1118, all of which may be directly or indirectly communicatively coupled to enable inter-component communication via a memory bus 1103, an I / O bus 1108, and an I / O bus interface 1110.

[0142] Computer system 1101 may include one or more general-purpose programmable central processing units (CPUs) 1102A, 1102B, 1102C, and 1102D, generally referred to herein as CPU 1102. In some embodiments, computer system 1101 may include a typical multiple processors of a relatively large system; however, in other embodiments, computer system 1101 may alternatively be a single-CPU system. Each CPU 1102 may execute instructions stored in memory subsystem 1104 and may include one or more levels of onboard cache. In some embodiments, the processor may include a memory controller and / or at least one or more memory controllers. In some embodiments, the CPU may execute the processes included herein (e.g., processes 400 and 500).

[0143] System memory subsystem 1104 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1122 or cache 1124. Computer system 1101 may also include other removable / non-removable, volatile / non-volatile computer system data storage media. By way of example only, storage system 1126 may be provided for reading from and writing to non-removable, non-volatile magnetic media such as a "hard disk drive". Although not shown, a disk drive may be provided for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk"), or an optical disc drive or other optical media may be provided for reading from or writing to a removable, non-volatile optical disc such as a CD-ROM, DVD-ROM, etc. Additionally, memory subsystem 1104 may include flash memory, such as a flash stick drive or flash drive. Memory devices may be connected to memory bus 1103 via one or more data media interfaces. Memory subsystem 1104 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments.

[0144] Although memory bus 1103 is Figure 7While shown as a single bus structure providing a direct communication path between CPU 1102, memory subsystem 1104, and I / O bus interface 1110, in some embodiments, memory bus 1103 may include multiple different buses or communication paths, which may be arranged in any of a variety of forms, such as hierarchical point-to-point links, star or mesh configurations, multi-layer buses, parallel and redundant paths, or any other suitable type of configuration. Furthermore, although I / O bus interface 1110 and I / O bus 1108 are shown as a single unit, in some embodiments, computer system 1101 may include multiple I / O bus interfaces 1110, multiple I / O buses 1108, or both. Additionally, although multiple I / O interface units separating I / O bus 1108 from various communication paths extending to individual I / O devices are shown, in other embodiments, some or all I / O devices may be directly connected to one or more system I / O buses.

[0145] In some embodiments, computer system 1101 may be a multi-user mainframe computer system, a single-user system, a server computer, or a similar device that has little or no direct user interface but receives requests from other computer systems (clients). Furthermore, in some embodiments, computer system 1101 may be implemented as a desktop computer, portable computer, laptop or notebook computer, tablet computer, pocket computer, telephone, smartphone, network switch or router, or any other suitable type of electronic device.

[0146] Notice, Figure 7 The description aims to depict representative main components of an exemplary computer system 1101. However, in some embodiments, the various components may have more... Figure 7 The greater or lesser complexity represented therein can exist differently from... Figure 7 The components shown or other components may vary in number, type, and configuration.

[0147] One or more programs / utilities 1128 may be stored in the memory subsystem 1104, each program / utility having at least one set of program modules 1130. Programs / utilities 1128 may include a hypervisor (also known as a virtual machine monitor), one or more operating systems, one or more applications, other program modules, and program data. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Programs / utilities 1128 and / or program modules 1130 generally perform the functions or methods of various embodiments.

[0148] From the above description, it will be understood that the proposed embodiments can provide one or more concepts for generating performance reports using an actual production system on which end users will run their integrations. An exemplary embodiment can be described as having two main parts:

[0149] (i) Collect performance results from the actual customers running the flow and assign the results to the specific integration type (i.e., the flow function) in the performance report; and

[0150] (ii) Use backup capacity in the production system by repeatedly looping through performance testing for each type of integration, and automatically ramping down once the capacity reaches a certain level.

[0151] Test results can be combined with the number of users / customers to allow reports to show the rates achieved by real customers for a given feature and / or the maximum achievable based on manually created tests. From such information, users can gain insights such as cost, latency, and maximum potential throughput. This allows users to evaluate the use of the plan and adjust, change, or match flows as needed. Furthermore, results can be logged based on message size and time of day to provide customers with additional information about how the performance of certain types of flows changes over the entire time period (e.g., a 24-hour period), thus enabling improved planning.

[0152] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0153] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0154] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0155] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0156] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0157] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0158] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0160] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0161] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “including” as used in this specification designate the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the preceding detailed description of exemplary embodiments of various embodiments, reference has been made to the accompanying drawings (in which like reference numerals denote like elements), which form part of the invention, and in which specific exemplary embodiments in which various embodiments may be practiced are illustrated by way of illustration. These embodiments have been described in sufficient detail to enable those skilled in the art to practice them, but other embodiments may be used, and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the preceding description, numerous specific details have been set forth to provide a thorough understanding of the various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the embodiments.

[0162] As used in this article, when referring to items, "multiple" means one or more items. For example, "multiple different types of networks" means one or more different types of networks.

[0163] When different reference numbers include a common number followed by different letters (e.g., 100a, 100b, 100c) or punctuation marks followed by different numbers (e.g., 100-1, 100-2 or 100.1, 100.2), the reference character (e.g., 100) used only when there are no letters or numbers following it can refer to a group of elements as a whole, any subset of that group, or an example sample of that group.

[0164] Furthermore, when used with a list of items, the phrase "at least one" means that different combinations of one or more of the listed items can be used, and it is possible that only one item from each item in the list is required. In other words, "at least one" refers to any combination of items, and any number of items from the list can be used, but not all items in the list are required. Items can be specific objects, things, or categories.

[0165] For example, but not limited to, "at least one of project A, project B, or project C" can include project A, project A and project B, or project B. This example could also include project A, project B and project C, or project B and project C. Of course, any combination of these projects can be provided. In some illustrative examples, "at least one" can be, for example, but not limited to, two projects A; one project B; and ten projects C; four projects B and seven projects C; or other suitable combinations.

[0166] Different instances of the term "embodiment" used in this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. Any data and data structures shown or described herein are merely examples, and in other embodiments, different amounts of data, data types, the number and types of fields, field names, the number and types of rows, records, entries, or organization of data may be used. Furthermore, any data can be logically combined, thus eliminating the need for a separate data structure. Therefore, the foregoing detailed description should not be construed as limiting.

[0167] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0168] Although the invention has been described with reference to specific embodiments, it is expected that changes and modifications of the invention will become apparent to those skilled in the art. Therefore, the appended claims should be construed as covering all such changes and modifications that fall within the true spirit and scope of the invention.

[0169] Embodiments of this disclosure provide a computer-implemented method for generating performance reports for a system suitable for running multiple streams, comprising: obtaining performance data associated with each of a plurality of running streams executed on the system; obtaining a plurality of templates, wherein each template describes the functionality of a stream; linking each of the plurality of running streams to one of the plurality of templates, the one template describing the functionality of the linked running stream; generating a performance summary for each of one or more templates based on the performance data associated with each of the plurality of running streams linked to that template; and generating a performance report based on the generated one or more performance summaries.

[0170] The method may further include generating a performance profile by: determining, for each of one or more templates, whether the number of runtimes linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtimes linked to the template satisfies the condition, generating a performance profile for each of the one or more templates based on performance data associated with each of the plurality of runtimes linked to the template.

[0171] The method may further include obtaining performance data by: for each of the plurality of running streams, obtaining one or more of the following: the latency of the running stream; the data rate per second of the running stream; the cost of the running stream; and the time to execute the running stream on the system.

[0172] The method may further include a performance profile comprising one or more of the following: an average value of the performance data of the plurality of running streams linked to the template; a range of the performance data of the plurality of running streams linked to the template; and a list of one or more performance data of the plurality of running streams linked to the template.

[0173] The method may further include linking each of the plurality of runtime flows by: for each of the plurality of runtime flows, determining the template among the plurality of templates that most closely describes the functionality of the runtime flow; and based on the determination, linking each of the plurality of runtime flows to one of the plurality of templates.

[0174] The method may further include each of a plurality of run streams being associated with one of a plurality of users, and each of the plurality of users being associated with one or more of the plurality of run streams. In some embodiments, the method may further include generating a performance report by: receiving a performance report query from a user among the plurality of users; generating the performance report based on one or more generated performance profiles, wherein each of the one or more performance profiles is based on performance data associated with one or more of the plurality of run streams, wherein at least one of the one or more run streams is associated with the user; and sending the performance report to the user in response to receiving the performance report query.

[0175] The method may also include a system that is a multi-tenant integrated cloud system, and each of the multiple run streams is allocated the predetermined processing capacity of the system.

[0176] Another embodiment of this disclosure provides a computer-implemented method for updating a performance report of a system adapted to run multiple streams, comprising: obtaining the performance report, wherein the performance report is based on one or more performance profiles associated with one or more of a plurality of templates; obtaining a plurality of test streams, wherein each test stream is associated with one of the plurality of templates; executing a set of the plurality of test streams on the system; obtaining test performance data associated with each of the test streams in the set of the plurality of test streams; generating a test performance profile for at least one of the plurality of templates based on the test performance data associated with each of the test streams associated with the template; and updating the performance report based on the generated one or more test performance profiles.

[0177] The method may further include obtaining test performance data by: for each of the plurality of test flows in the set, obtaining one or more of the following: the latency of the test flow; the data rate per second of the test flow; the cost of the test flow; and the time to execute the test flow on the system.

[0178] The method may further include, wherein executing the set of the plurality of test flows comprises: determining the set of the plurality of test flows based on the idle processing capacity of the system; and executing the set of the plurality of test flows on the system.

[0179] The method may further include determining the set of the plurality of test flows, which further includes: determining the idle processing capacity of the system; increasing the amount of system processing capacity used by the test flows in response to determining that the idle processing capacity of the system is higher than a predetermined capacity threshold; and decreasing the amount of system processing capacity used by the test flows in response to determining that the idle processing capacity of the system is higher than the predetermined capacity threshold.

[0180] The method may further include increasing the processing capacity of the system used by the test stream by at least one of the following: adding one or more test streams to a set of the plurality of test streams; and increasing the data throughput of one or more of the test streams in the set of the plurality of test streams, and wherein decreasing the processing capacity of the system used by the test stream by at least one of the following: removing one or more test streams from the set of the plurality of test streams; and decreasing the data throughput of one or more of the test streams in the set of the plurality of test streams.

[0181] The method may further include generating a performance report for a system suitable for running multiple streams, and further includes: updating the performance report, wherein the execution of the set of multiple test streams includes: determining the set of multiple test streams based on the idle processing capacity of the system; and executing the set of multiple test streams on the system.

[0182] Another embodiment of this disclosure provides an apparatus for generating performance reports for a system suitable for running multiple streams, comprising: a measurement unit configured to obtain performance data associated with each of the plurality of running streams; a mapping unit configured to: obtain a plurality of templates, wherein each template describes the functionality of a stream; and link each of the plurality of running streams to one of the plurality of templates, the template describing the functionality of the linked running stream; and a report generator configured to: generate a performance summary for each of one or more templates based on the performance data associated with each of the plurality of running streams linked to the template; and generate a performance report based on the generated one or more performance summaries.

[0183] The apparatus may also include a report generator configured to: for each of one or more templates, determine whether the number of runtime streams linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtime streams linked to the template satisfies the condition, generate a performance summary for each of the one or more templates based on performance data associated with each of the plurality of runtime streams linked to the template.

[0184] The device may also include a metric unit configured to obtain one or more of the following for each of a plurality of running streams: the latency of the running stream; the data rate per second of the running stream; the cost of the running stream; and the time to execute the running stream on the system.

[0185] The apparatus may also include a report generator configured to: receive a performance report query from a user; generate the performance report based on one or more generated performance profiles, each of the one or more performance profiles being based on performance data associated with one or more of the plurality of running streams, some of the one or more running streams being associated with the user; and transmit the performance report to the user in response to receiving the performance report query.

[0186] The device may further include a system that is a multi-tenant integrated cloud system, and a predetermined processing capacity limit for each of the multiple running streams assigned to the system.

[0187] Another embodiment of this disclosure provides an apparatus for updating a performance report of a system adapted to run multiple streams, comprising: a report generator configured to: obtain the performance report, wherein the performance report is based on one or more performance profiles associated with one or more templates of a plurality of templates; a performance test driver configured to: obtain a plurality of test streams, wherein each test stream is associated with one of the plurality of templates; drive the execution of a set of the plurality of test streams on the system; and a metric unit configured to: obtain test performance data associated with each of the plurality of executed test streams; and wherein the report generator is further configured to: generate a test performance profile for at least one of the plurality of templates based on the test performance data associated with each of the test streams associated with the template; and update the performance report based on the generated one or more test performance profiles.

[0188] The device may further include the metric unit configured to obtain one or more of the following for each of the plurality of test streams: the latency of the test stream; the data rate per second of the test stream; the cost of the test stream; and the time to execute the test stream on the system.

[0189] The apparatus may also include a performance test driver configured to: determine the set of the plurality of test streams based on the idle processing capacity of the system; and drive the set of the plurality of test streams to be executed on the system.

[0190] The apparatus may also include a performance test driver further configured to: determine the idle processing capacity of the system; increase the amount of system processing capacity used by the test stream in response to determining that the idle processing capacity of the system is higher than a predetermined capacity threshold; and decrease the amount of system processing capacity used by the test stream in response to determining that the idle processing capacity of the system is higher than the predetermined capacity threshold.

[0191] The apparatus may also include a performance test driver configured to: add one or more test streams to a set of the plurality of test streams; increase the data throughput of one or more of the test streams in the set of the plurality of test streams; remove one or more test streams from the set of the plurality of test streams; and decrease the data throughput of one or more of the test streams in the set of the plurality of test streams.

[0192] Embodiments of this disclosure also provide an apparatus for generating performance reports for a system configured to run multiple streams. The apparatus includes a metric unit configured to obtain performance data associated with each of the plurality of running streams. The apparatus further includes a mapping unit configured to: obtain a plurality of templates, each template describing the functionality of a stream; and link each of the plurality of running streams to one of the templates, the template describing the functionality of the linked running stream; and a report generator configured to: generate a performance summary for each of one or more templates based on the performance data associated with each of the plurality of running streams linked to the template; and generate a performance report based on the generated one or more performance summaries.

[0193] According to some embodiments, the report generator may be further configured to: for each of one or more templates, determine whether the number of runtime streams linked to the template satisfies a condition based on a predetermined number; and in response to determining that the number of runtime streams linked to the template satisfies the condition, for each of the one or more templates, generate a performance summary based on performance data associated with each of the plurality of runtime streams linked to the template.

[0194] According to some embodiments, the metric unit can be configured to obtain one or more of the following for each of the plurality of running flows: the waiting time of the running flow; the data rate per second of the running flow; the cost of the running flow; and the time to execute the running flow on the system.

[0195] According to some embodiments, the report generator may be further configured to: receive a performance report query from a user; generate the performance report based on one or more generated performance profiles, each of the one or more performance profiles being based on performance data associated with one or more of the plurality of running flows, some of the one or more running flows being associated with the user; and transmit the performance report to the user in response to receiving the performance report query.

[0196] According to some embodiments, the system may be a multi-tenant integrated cloud system, wherein each of the multiple running streams is assigned a predetermined processing capacity limit of the system.

[0197] According to some embodiments of this disclosure, an apparatus is provided for updating a performance report of a system configured to run multiple streams. The apparatus includes a report generator configured to obtain a performance report based on one or more performance summaries associated with one or more of a plurality of templates. The apparatus also includes a performance test driver configured to: obtain a plurality of test streams, each test stream associated with one of the plurality of templates; and drive the execution of a set of the plurality of test streams on the system; and a metric unit configured to: obtain test performance data associated with each of the plurality of executed test streams. The report generator is further configured to generate a test performance summary for at least one of the plurality of templates based on the test performance data associated with each of the test streams associated with the template; and to update the performance report based on the generated one or more test performance summaries.

[0198] According to some embodiments, the metric unit can be configured to obtain one or more of the following for each of the plurality of test streams: the latency of the test stream; the data rate per second of the test stream; the cost of the test stream; and the time to execute the test stream on the system.

[0199] According to some embodiments, a performance test driver can be configured to: determine the set of the plurality of test streams based on the idle processing capacity of the system; and drive the execution of the set of the plurality of test streams on the system.

[0200] According to some embodiments, the performance test driver may be further configured to: determine the idle processing capacity of the system; increase the amount of processing capacity of the system used by the test stream in response to determining that the idle processing capacity of the system is higher than a predetermined capacity threshold; and decrease the amount of processing capacity of the system used by the test stream in response to determining that the idle processing capacity of the system is lower than a predetermined capacity threshold.

[0201] According to some embodiments, the performance test driver may be further configured to: add one or more test streams to a set of the plurality of test streams; increase the data throughput of one or more of the test streams in the set of the plurality of test streams; remove one or more test streams from the set of the plurality of test streams; and decrease the data throughput of one or more of the test streams in the set of the plurality of test streams.

Claims

1. A computer-implemented method for generating performance reports for a system configured to run multiple streams, wherein a stream refers to a set of actions connected together to perform a task, comprising: Obtain performance data associated with each of a plurality of run streams on the system, wherein each of the plurality of run streams includes a connected sequence of nodes, wherein each node in the connected sequence of nodes is configured to perform the task, and wherein each of the plurality of run streams is associated with one of a plurality of users; Obtain multiple templates, each of which describes the functionality of the flow; Each of the plurality of execution flows is linked to a template in the plurality of templates that describes the functionality of the linked execution flow; For each of one or more templates describing the functionality of the linked run streams, a performance profile is generated based on performance data associated with each of the plurality of run streams linked to the template, wherein if the number of run streams linked to the template does not meet a condition based on a predetermined number, a performance profile is not generated for the template. as well as Generate a performance report based on one or more generated performance summaries, wherein generating the performance summaries further includes: For each of the one or more templates, determine the number of runtime streams linked to the template that satisfy a condition based on a predetermined number; as well as For each of the one or more templates, the performance profile is generated based on the performance data associated with each of the plurality of runtime streams linked to the template.

2. The computer-implemented method according to claim 1, wherein obtaining performance data includes: For each of the plurality of execution flows, obtain one or more selected from the group consisting of: The waiting time of the running stream; The data rate per second of the running stream; The cost of the operation flow; and The time it takes for the runtime flow to be executed on the system.

3. The computer-implemented method of claim 1, wherein the performance profile includes one or more selected from the group consisting of: The average of the performance data of the multiple running streams linked to the template; The range of performance data linked to the multiple runtime streams of the template; as well as A list of one or more of the performance data from the multiple running streams linked to the template.

4. The computer-implemented method of claim 1, wherein linking each of the plurality of execution streams comprises: For each of the plurality of execution flows, determine the template that most closely describes the function of the execution flow among the plurality of templates; as well as Based on the determination, each of the plurality of runtime flows is linked to one of the plurality of templates.

5. The computer-implemented method according to claim 1, wherein generating the performance report further includes: Receive performance report queries from one of the multiple users; The performance report is generated based on one or more performance profiles, each of which is based on performance data associated with one or more of the plurality of run streams, and at least one of the plurality of run streams is associated with the user of the plurality of users. as well as In response to receiving the performance report query, the performance report is sent to the user among the plurality of users.

6. The computer-implemented method of claim 1, wherein the system is a multi-tenant integrated cloud system, and wherein each of the plurality of running streams is allocated a predetermined processing capacity of the system.

7. The computer-implemented method according to claim 1, further comprising: The identification is that the first running flow among the plurality of running flows has a first function that is not described in the plurality of templates, which makes it impossible for the first running flow to be linked to one of the plurality of templates; as well as Generate a new template that describes the first function of the first execution flow, wherein the new template describes the first function of the first execution flow and is generated by summarizing an existing template.

8. The computer-implemented method of claim 1, wherein the condition requires linking two or more of the running streams to each of the one or more templates to generate the performance profile.

9. A computer-implemented method for updating a performance report of a system configured to run multiple streams, the performance report being generated for the system using the method according to any one of claims 1-8, comprising: Obtain a performance report, wherein the performance report is based on one or more performance summaries associated with one or more of multiple templates; Obtain multiple test streams, where each test stream is associated with one of the multiple templates; Execute the set of multiple test flows on the system; Obtain test performance data associated with each of the multiple test streams in the set; For at least one of the plurality of templates, a test performance summary is generated based on test performance data associated with each of the test streams associated with the template; as well as The performance report is updated based on one or more test performance summaries generated.

10. The computer-implemented method according to claim 9, wherein obtaining test performance data includes: For each test flow in the set of multiple test flows, obtain one or more from a group consisting of the following: The waiting time for the test stream; The data rate per second of the test stream; The cost of the test flow; as well as The time it takes to execute the test stream on the system.

11. The computer-implemented method of claim 9, wherein executing the set of the plurality of test flows comprises: The set of the multiple test streams is determined based on the system's idle processing capability; as well as The set of multiple test flows is executed on the system.

12. The computer-implemented method of claim 11, wherein determining the set of the plurality of test streams further comprises: Determine the idle processing capacity of the system; In response to determining that the idle processing capacity is higher than a predetermined capacity threshold, the amount of processing capacity of the system used by the test stream is increased; as well as In response to determining that the idle processing capacity is below a predetermined capacity threshold, the amount of processing capacity of the system used by the test stream is reduced.

13. The computer-implemented method of claim 12, wherein the amount by which the processing power of the system used by the test stream is increased comprises at least one selected from the group consisting of: Add one or more test flows to the set of multiple test flows; as well as Increase the data throughput of one or more of the test streams in the set of multiple test streams, and The amount by which the processing power of the system used by the test stream is reduced includes at least one of the following: Remove one or more test flows from the set of multiple test flows; as well as Reduce the data throughput of one or more of the test streams in the set of multiple test streams.

14. A system comprising: processor; as well as A computer-readable storage medium communicatively coupled to the processor and storing program instructions that, when executed by the processor, cause the processor to perform a method for generating a performance report for a system configured to run multiple streams, according to any one of claims 1-8.

15. A system comprising: processor; as well as A computer-readable storage medium communicatively coupled to the processor and storing program instructions that, when executed by the processor, cause the processor to perform a method for updating a performance report of a system configured to run multiple streams, according to any one of claims 9-13.

16. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to perform a method for generating a performance report for a system configured to run multiple streams according to any one of claims 1-8.

17. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to perform a method for updating a performance report of a system configured to run multiple streams according to any one of claims 9-13.

Images