Performance verification device

Abstract

This invention provides a performance verification device that verifies in advance whether an application will perform as desired. [Solution] The performance verification device 100 for performing benchmarking of a low-code platform includes a test execution unit 1105 that collects information on the correlation between multiple flows by executing multiple flows deployed on the low-code platform 300 at least partially simultaneously, performs load tests on the multiple flows, and collects the test results of the load tests; and a benchmark calculation unit 1140 that identifies bottlenecks in the multiple flows based on the test results and information on the correlations, and calculates a benchmark for the performance of the low-code platform from the bottlenecks of the multiple flows.

Description

Technical Field

[0001] The present invention relates to a performance verification device, and is suitable for application to a performance verification device related to a technology for verifying the performance of an application, for example.

Background Art

[0002] In recent years, a development platform called a low-code application platform (hereinafter abbreviated as "LCAP") for developing applications without coding programs has emerged. In such a development platform using LCAP, it is difficult to predict how much performance can be exhibited with respect to desired specifications. Patent Document 1 discloses a technique for identifying an item that becomes a performance bottleneck (hereinafter referred to as a "bottleneck item") by collecting an execution log of an application and estimating the bottleneck and load characteristics of the application.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique disclosed in Patent Document 1, although the bottleneck item can be estimated when the load configuration of the application is clearly known, when the description content of the application is not clearly understood, the bottleneck item cannot be identified unless the execution log of the application contains detailed content. Therefore, with the technique disclosed in Patent Document 1, it was not possible to verify in advance whether the application would exhibit desired performance when actually operating.

[0005] This invention was made in consideration of the above points, and aims to propose a performance verification device that can verify in advance whether or not an application will perform as desired. [Means for solving the problem]

[0006] To solve these problems, the present invention provides a performance verification device for benchmarking a low-code platform, comprising: a test execution unit that collects information on the correlation between a plurality of flows deployed on the low-code platform by executing the plurality of flows at least partially simultaneously, performs load tests on the plurality of flows, and collects the test results of the load tests; and a benchmark calculation unit that identifies the bottlenecks of the plurality of flows based on the test results and the information on the correlations, and calculates a benchmark for the performance of the low-code platform from the bottlenecks of the plurality of flows. [Effects of the Invention]

[0007] According to the present invention, it is possible to verify in advance whether or not an application will perform as desired. [Brief explanation of the drawing]

[0008] [Figure 1] This system configuration diagram shows an example of the overall configuration of an information processing system including a performance verification device according to this embodiment. [Figure 2] This figure shows an example of a test flow. [Figure 3] This figure shows an example of test flow information. [Figure 4] This figure shows an example of benchmark item information. [Figure 5] This figure shows an example of primary estimation information for load items. [Figure 6] This figure shows an example of test result information. [Figure 7] This figure shows an example of inter-flow correlation information. [Figure 8]This figure shows an example of bottleneck information. [Figure 9] This figure shows an example of benchmark result information. [Figure 10] This flowchart shows an example of the procedure for the main processing according to this embodiment. [Figure 11] Figure 10 is a flowchart illustrating an example of the correlation measurement process. [Figure 12] Figure 10 is a flowchart illustrating an example of the benchmark calculation process. [Figure 13] This figure shows an example of a benchmark results screen. [Modes for carrying out the invention]

[0009] Hereinafter, this embodiment of the present invention will be described in detail with reference to the drawings. Figure 1 is a system configuration diagram showing an example of the overall configuration of an information processing system including a performance verification device 100 according to this embodiment. The information processing system according to this embodiment comprises a low-code platform 300 and a performance verification device 100, and preferably further comprises a user client 200.

[0010] The Low-Code Platform 300 is a development platform that allows software development with minimal program coding by, for example, using a GUI (Graphical User Interface) to connect multiple objects with predetermined functions and setting some input / output. Such software does not include programs that require a very large amount of coding.

[0011] The performance verification device 100 conducts benchmarks on the low-code base 300. The performance verification device 100 includes a memory 1000, a CPU (Central Processing Unit) 2000, a storage device 3000, an input / output interface (abbreviated as "I / F") 4000, a network I / F 5000, and a bus 500. These memory 1000, CPU 2000, storage device 3000, input / output I / F 4000, and network I / F 5000 are connected to each other via the bus 500.

[0012] The memory 1000 has a processing unit 1100 and a data management unit 1200. The processing unit 1100 has a test execution unit 1105, a benchmark calculation unit 1140, and a load item primary estimation unit 1130. Note that the load item primary estimation unit 1130 may be a part of the benchmark calculation unit 1140.

[0013] The data management unit 1200 has test flow information 1210, benchmark item information 1220, load item primary estimation information 1230, test result information 1240, inter-flow correlation information 1250, bottleneck information 1260, and benchmark result information 1270. In the illustrated example, for the purpose of clearly showing the input / output relationship of each data, it is shown as separate data. However, for example, the test flow information 1210, the load item primary estimation unit 1130, and the test result information 1240 may be data in the same table format.

[0014] The test flow information 1210 includes information regarding the test flow for the application to be tested. The information regarding the test flow includes, for example, a script as descriptive content for defining the functions of the application. Here, it is unknown which part of the script included in the application becomes a performance bottleneck.

[0015] The benchmark item information 1220 indicates information regarding the benchmark items (also referred to as "benchmark items") for the test to be conducted. Details of the benchmark item information 1220 will be described later.

[0016] The load item primary estimation information 1230 indicates information estimated as the benchmark item that is the load. The load item primary estimation information 1230 represents, for each test flow, the possibility of having the load of the benchmark item by a numerical value, for example, between 0.0 and 1.0.

[0017] The test result information 1240 is the test result of the load test executed by the test execution unit 1105. The inter-flow correlation information 1250 is information indicating the correlation between a plurality of flows. The bottleneck information 1260 is information regarding the bottleneck when executing a plurality of flows. The benchmark result information 1270 is information indicating the result of the benchmark based on the test result.

[0018] The test execution unit 1105 executes a plurality of flows deployed on the low-code base, at least partially simultaneously, for example, in parallel, concurrently, or simultaneously, to collect the inter-flow correlation information 1250 as an example of information regarding the correlation between a plurality of flows, perform a load test on a plurality of flows, and collect the test results of this load test. The test execution unit 1105 includes a test measurement unit 1110 and a correlation measurement unit 1120.

[0019] The test measurement unit 1110 executes each test flow, measures its test result, and records it. Specifically, the test measurement unit 1110 performs tests, for example, on a plurality of test flows.

[0020] The correlation measurement unit 1120 measures the correlation regarding a plurality of test flows. Specifically, the correlation measurement unit 1120 measures whether there is a correlation between a plurality of test flows for testing the target application.

[0021] The load item primary estimation unit 1130 uses the test flow information 1210 to generate load item primary estimation information 1230, which is information that estimates which benchmark items will be subjected to load when each flow is executed. Note that the load item primary estimation unit 1130 may be part of the functions of the benchmark calculation unit 1140.

[0022] The benchmark calculation unit 1140 identifies bottlenecks in multiple flows based on the test results and inter-flow correlation information 1250 described above, and calculates a benchmark for the performance of the low-code platform 300 from the bottlenecks in the multiple flows.

[0023] The benchmark calculation unit 1140 estimates the load configuration of each of the multiple flows from a script which is an example of the description content of the multiple flows, and identifies the bottlenecks of the multiple flows as test results based on the load configuration of each of the multiple flows.

[0024] The benchmark calculation unit 1140 selects the benchmark item with the highest likelihood calculated from information on the correlation between multiple flows and the load configuration, from among the benchmark items that are candidates for bottlenecks in multiple flows.

[0025] The benchmark calculation unit 1140 estimates the lower limit of the benchmark result from the test results and bottlenecks in multiple flows.

[0026] The benchmark calculation unit 1140 has information on benchmark items as an example of prior information used to estimate the load configuration, and updates the prior information from the process of identifying bottlenecks in multiple flows and the test results.

[0027] The benchmark calculation unit 1140 receives supplemental information about multiple flows from an external source, such as the user of the performance verification device 100, including, for example, information obtained from a model or specification and at least one of the information provided by the user, and identifies bottlenecks in the multiple flows based on the test results and supplemental information. This supplemental information may include, for example, a description by the user.

[0028] The benchmark calculation unit 1140 may, for example, estimate the performance of flows that may be adopted in the future based on the calculated benchmark results.

[0029] The test execution unit 1105 modifies at least one of the multiple flows based on the calculated benchmark results, collects information again about the correlation between the multiple flows including the modified flow, and collects test results again that include information about the correlation between the multiple flows.

[0030] The user client 200 is a client computer operated by a user utilizing the performance verification device 100 described above, and is not, for example, a client computer operated by a user utilizing the low-code platform 300.

[0031] Figure 2 shows an example of a test flow 199. The test flow 199 has, for example, two tasks 103 and 105. Specifically, in the test flow 199, for example, the first task 103 receives data 102 from the starting point 101. In the example shown, the low-code platform 300 is the primary target, but services that allow applications to be developed and executed in small function units without graphical programming, such as FaaS (Function as a Service), can also be targeted.

[0032] The first task 103 processes data 102 as input and outputs data 104 to the second task 105. The second task 105 processes data 104 as input and outputs data 106 to the termination point 107.

[0033] Figure 3 shows an example of test flow information 1210. The test flow information 1210 manages columns such as, for example, flow ID (IDentifier) ​​1211, name 1212, and definition content 1213.

[0034] In this test flow information 1210, the test flow with test flow ID "FL1" is named "DB Write" and corresponds to a database write flow, managing the definition content which is written by a script.

[0035] Furthermore, in this test flow information 1210, for example, the test flow with test flow ID "FL2" is named "PDF Output," which corresponds to a PDF file output flow, and the definition content, which is the content written by the script, is managed.

[0036] Figure 4 shows an example of benchmark item information 1220. Benchmark item information 1220 manages the following columns: item ID (IDentifier) ​​1221, name 1222, and summary 1223. In benchmark item information 1220, for example, the name of the benchmark item for which item ID 1221 is "BM1" is managed as "flow initialization".

[0037] Figure 5 shows an example of the primary estimation information 1230 for load items. The tabular data shown represents the probability that each test flow has the computational load for each benchmark item. Note that such probabilities may also be estimated using API (Application Programming Interface) technology or machine learning technology.

[0038] The primary load item estimation information 1230 manages each benchmark item ID for each flow ID (IDentifier). For example, if the flow ID is "FL1", the primary load item estimation information 1230 indicates that the benchmark item "BM1" is 0.9, the benchmark "BM2" is 0.1, the benchmark "BM3" is 0.9, and the benchmark item "BM4" is 0.1.

[0039] The primary load item estimation information 1230 represents the likelihood of each test flow having a computational load for each benchmark item as a numerical value between 0.0 and 1.0. For example, a value closer to 1.0 indicates a higher probability of a load being incurred during flow execution, while a value closer to 0.0 indicates a lower probability of no load being incurred.

[0040] Figure 6 shows an example of test result information 1240. Test result information 1240 manages columns such as flow ID 1241, response time 1242, and maximum throughput 1243. Test result information 1240 shows the test results for each flow ID by the test measurement unit 1110. For example, if the flow ID is "FL1", test result information 1240 shows that the response time 1242 is 10 seconds and the maximum throughput is 15 per second. In other words, test result information 240 manages the response time 1242 and maximum throughput 1243 for each flow ID.

[0041] Figure 7 shows an example of inter-flow correlation information 1250. Inter-flow correlation information 1250 represents, for example, the relative correlation between two flows in tabular data. In inter-flow correlation information 1250, the (i,j) component of the table represents, with logical values ​​of 0 and 1, whether flow j has a benchmark item that is a bottleneck for flow i. Inter-flow correlation information 1250 is measured by the correlation measurement unit 1120. Note that diagonal components where the flow ID on the vertical axis and the flow ID on the horizontal axis are the same are not measured because the information itself is meaningless.

[0042] In the example shown, for instance, the flow ID "FL2" on the horizontal axis corresponds to "FL3" on the vertical axis, and the flow ID "FL3" on the horizontal axis corresponds to "FL2" on the vertical axis, and both values ​​are "0," so it is determined that the correlation between the two flows indicated by the two flow IDs is low.

[0043] On the other hand, for example, if the flow ID "FL3" on the horizontal axis corresponds to the flow ID "FL1" on the vertical axis, and the flow ID "FL1" on the horizontal axis corresponds to the flow ID "FL3" on the vertical axis, and both values ​​are the same at "1", then it is determined that there is a high correlation between the two flows indicated by the two flow IDs.

[0044] Figure 8 shows an example of bottleneck information 1260. Bottleneck information 1260 manages columns such as flow ID 1261 and bottleneck 1262.

[0045] The example diagram illustrates which benchmark item is limiting the throughput of each flow, or which is causing the bottleneck.

[0046] Figure 9 shows an example of benchmark result information 1270. Benchmark result information 1270 represents the benchmark results of a performance verification device 100 for a specific low-code platform 300. In this embodiment, the benchmark targets are mainly throughput, for example, but response performance may also be included.

[0047] Benchmark result information 1270 shows, for example, the maximum throughput 1272 for each benchmark item ID (IDentifier) ​​1271. By referring to benchmark result information 1270, you can find out what the maximum throughput is for each benchmark item.

[0048] The information processing system, including the performance verification device 100, has the configuration described above. Next, an example of the operation of the performance verification device 100 will be explained.

[0049] Figure 10 is a flowchart showing an example of the procedure for the main processing according to this embodiment. In step S101, the test measurement unit 1110 collects test result information 1240 using test flow information 1210.

[0050] In step S102, the correlation measurement unit 1120 performs correlation measurement processing. In this correlation measurement processing, the correlation measurement unit 1120 collects inter-flow correlation information 1250 using the test flow information 1210.

[0051] In step S103, the load item primary estimation unit 1130 uses the test flow information 1210 to generate load item primary estimation information 1230 as information estimated to be benchmark items that are loads.

[0052] In step S104, the benchmark calculation unit 1140 executes the benchmark calculation process. In this benchmark calculation process, the benchmark calculation unit 1140 performs calculations using the collected and specified information and obtains bottleneck information 1260 and benchmark result information 1270.

[0053] Figure 11 is a flowchart showing an example of the correlation measurement process procedure shown in Figure 10. In step S201, the correlation measurement unit 1120 repeatedly executes steps S202 to S210 for all test flows i (where i is an integer) (step S211).

[0054] In step S202, the correlation measurement unit 1120 performs a load test by repeatedly sending requests to the low-code platform 300 (corresponding to LCAP in the diagram) using the test flow i.

[0055] In step S203, the correlation measurement unit 1120 gradually increases the request frequency, and if an impact is observed on the measured response time, it maintains the request frequency at that point.

[0056] In step S204, the correlation measurement unit 1120 repeatedly executes steps S205 to S208 and S209 for all flows except flow i for test flow j (step S210). Note that test flow j ≠ test flow i.

[0057] In step S205, the correlation measurement unit 1120 collects the normal response time t1 of test flow j from the test result information 1240. In step S206, the correlation measurement unit 1120 executes test flow j a certain number of times and measures the response time t2.

[0058] Furthermore, a step may be added between steps S4 and S5 described above to record the frequency of requests at that time in the test result information 1240. Also, the processing by the test measurement unit 1110 and the processing by the correlation measurement unit 1120 may be performed simultaneously.

[0059] In step S206, the correlation measurement unit 1120 determines whether the difference between response time t1 and response time t2 exceeds a certain threshold. If the difference between response time t1 and response time t2 exceeds a certain threshold, step S208 is executed. If the difference between response time t1 and response time t2 does not exceed a certain threshold, step S209 is executed.

[0060] In step S208, the correlation measurement unit 1120 records "1" in the i row and j column of the inter-flow correlation information 1250. On the other hand, in step S209, the correlation measurement unit 1120 records "0" in the i row and j column of the inter-flow correlation information 1250. As a result of this recording, the inter-flow correlation information 1250 will be as shown in Figure 7.

[0061] Figure 12 is a flowchart showing an example of the procedure for the benchmark calculation process shown in Figure 10. In step S301, the benchmark calculation unit 1140 repeatedly executes the following steps S302 to S313 for all test flows i (where i is an integer) (step S314).

[0062] In step S302, the benchmark calculation unit 1140 repeatedly performs the following actions on the low-code platform 300 (corresponding to LCAP in the diagram) in response to requests using the benchmark item m test flow i.

[0063] In step S303, the benchmark calculation unit 1140 calculates the probability p that the bottleneck in test flow i is benchmark item m. i,m Initialize it to "1".

[0064] In step S304, the benchmark calculation unit 1140 calculates the probability p i,m The i and m components of the primary load item estimation information 1230 are multiplied by this. In step S305, the benchmark calculation unit 1140 extracts flow j, which has a value of "1" in row i, from the inter-flow correlation information 1250.

[0065] In step S306, the benchmark calculation unit 1140 determines whether the j and i components of the inter-flow correlation information 1250 are "1". If the j and i components of the inter-flow correlation information 1250 are "1", step S307 is executed; however, if the j and i components of the inter-flow correlation information 1250 are not "1", step S309 is executed.

[0066] In step S307, the benchmark calculation unit 1140 multiplies the probability p(i,m) by the j and m components of the primary load item estimation information 1230. In step S308, the benchmark calculation unit 1140 checks from the test result information 1240 whether the difference between test flow i and test flow j is within a predetermined threshold.

[0067] Meanwhile, in step S309, the benchmark calculation unit 1140 multiplies the probability p(i,m) by the j and m components of the primary load item estimation information 1230.

[0068] In step S310, as described above, the benchmark calculation unit 1140 repeatedly executes steps S305 to S310 for the other extracted flows. In step S311, the benchmark calculation unit 1140 repeatedly executes steps S302 to S310 for all benchmark items.

[0069] Note that the results may differ depending on the iteration order described above, so you may perform calculations multiple times with different orders and adopt the one with the highest overall likelihood. Also, if there are items for which the maximum throughput has not been estimated in the above process, you may extract flows that are likely to contain that item and estimate their maximum throughput value using the lower limit of that item (image of the value of "BM2" shown in Figure 9).

[0070] Figure 13 shows an example of the benchmark results screen 298. The benchmark results screen 298 is a screen displayed by the benchmark calculation unit 1140, mainly based on the benchmark result information 1270. In other words, the benchmark results screen 298 displays the results of benchmarking multiple low-code platforms 300. The benchmark results screen 298 may also include screens on other tabs, for example, to check test flows and test flow execution results.

[0071] The benchmark results screen 298 has low-code board name display fields 201, 202, and 203 that show the names of the low-code boards 300, and performance display fields 207 to 212 that show the performance values ​​of each low-code board. Each performance display field 207 to 212 displays the performance value for the corresponding low-code board.

[0072] For example, performance display area 207 displays performance values ​​related to the flow initialization process that initializes the test flow. Performance display area 208 displays performance values ​​related to authentication. Performance display area 209 displays performance values ​​related to the script call process that calls each script that describes the test flow. Performance display area 210 displays performance values ​​related to the DB write process when writing to the database. Performance display area 211 displays performance values ​​related to the DB read process when reading to the database. Performance display area 212 displays performance values ​​related to external communication.

[0073] On the benchmark results screen 298, the bottleneck performance value 299, which is identified as a bottleneck based on the benchmark results, is displayed, for example, in a shaded display format. The bottleneck performance value 299 indicates a performance value that may be a bottleneck, as determined by the benchmark calculation unit 1140 based on the benchmark result information 1270.

[0074] The example in the diagram shows that the performance value of script calls on the low-code platform "LCAP A" is "20 / s," which may exceed a predetermined threshold. Comparing the performance values ​​of script calls, the example in the diagram shows that the low-code platform "LCAP A" performs worse in benchmarks than, for example, two other low-code platforms, "LCAP B" and "LCAP C," indicating that it could become a performance bottleneck.

[0075] The performance verification device 100 according to this embodiment performs benchmarking of the low-code platform 300 and includes a test execution unit 1105 that executes multiple flows deployed on the low-code platform 300 at least partially simultaneously to collect inter-flow correlation information 1250 as an example of information regarding the correlation between multiple flows, performs load tests on the multiple flows and collects the test results of these load tests, and a benchmark calculation unit 1140 that identifies bottlenecks in the multiple flows based on these test results and the inter-flow correlation information 1250, and calculates a benchmark for the performance of the low-code platform 300 from the bottlenecks of the multiple flows.

[0076] With this configuration, load tests are performed on multiple flows while considering the correlations between them, allowing for the accurate identification of bottlenecks in multiple flows based on those correlations. Therefore, it is possible to verify in advance and more accurately whether the low-code platform 300, as an example application, will achieve the desired performance. Furthermore, since there is no need to assign a dedicated system engineer to the aforementioned verification, costs can be reduced.

[0077] In this embodiment, the benchmark calculation unit 1140 estimates the load configuration of multiple flows from a script, which is an example of a description of the contents of multiple flows, and identifies the bottlenecks of the multiple flows as test results based on the load configuration of each flow. In this way, because the bottlenecks identified based on the load configuration estimated from the script are identified, it is possible to verify in advance and more accurately whether the application will achieve the desired performance.

[0078] In this embodiment, the benchmark calculation unit 1140 selects the benchmark item with the highest likelihood calculated from information on the correlation between multiple flows and the load configuration, from among benchmark items (for example, "Script Call" in Figure 13) that are candidates for bottlenecks in multiple flows (for example, the group of performance values ​​displayed in the performance display section 209 of the script call in Figure 13). In this way, since the benchmark item is selected according to the likelihood calculated from information on the correlation between multiple flows and the load configuration, it is possible to verify in advance and more accurately whether the application will achieve the desired performance.

[0079] In this embodiment, the benchmark calculation unit 1140 estimates a lower limit of the benchmark result from the test results and bottlenecks in multiple flows. By doing so, it is possible to verify in advance and more accurately whether the application will achieve the desired performance, since the lower limit of the benchmark result is estimated.

[0080] In this embodiment, the benchmark calculation unit 1140 has information on benchmark items as an example of prior information used to estimate the load configuration, and updates the prior information from the process of identifying bottlenecks in multiple flows and the test results. In this way, it is possible to verify in advance and more accurately whether the application will achieve the desired performance based on the updated prior information.

[0081] In this embodiment, the benchmark calculation unit 1140 receives supplementary information about multiple flows from an external source and identifies bottlenecks in the multiple flows based on the test results and supplementary information. In this way, bottlenecks are identified by referring to the supplementary information, making it possible to verify in advance and more accurately whether the application will achieve the desired performance.

[0082] In this embodiment, the benchmark calculation unit 1140 may, for example, estimate the performance of a flow that may be adopted in the future based on the benchmark results. This makes it possible to verify in advance and more accurately whether an application including a flow that may be adopted in the future will achieve the desired performance.

[0083] In this embodiment, the test execution unit 1105 modifies at least one of the multiple flows based on the calculated benchmark results, collects information on the correlation between the multiple flows including the modified flow, and collects test results including the information on the correlation between the multiple flows again. In this way, it is possible to verify in advance and more accurately whether the application will perform as desired based on the newly collected test results.

[0084] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the configurations described. Also, each element described in parallel in this embodiment may be configured such that at least one of the elements is connected in series with respect to the other elements. [Industrial applicability]

[0085] The present invention can be applied to a performance verification device relating to a technique for verifying the performance of an application. [Explanation of Symbols]

[0086] 1110...Test measurement unit, 1150...Test execution unit, 1120...Correlation measurement unit, 1140...Benchmark calculation unit, 1210...Test flow information, 1220...Benchmark item information, 1230...Load item primary estimation information, 1240...Test result information, 1250...Inter-flow correlation information, 1260...Bottleneck information, 1270...Benchmark result information.

Claims

1. A performance verification device for conducting benchmarks on a low-code platform, A test execution unit that collects information regarding the correlation between multiple flows by executing multiple flows deployed on the low-code platform at least partially simultaneously, performs load tests on the multiple flows, and collects the test results of the load tests, A benchmark calculation unit identifies the bottlenecks of the multiple flows based on the test results and the correlation information, and calculates a benchmark for the performance of the low-code platform from the bottlenecks of the multiple flows. A performance verification device characterized by comprising the following features.

2. The benchmark calculation unit, Based on the descriptions of the multiple flows, the load configuration of each of the multiple flows is estimated, and based on the load configuration of each of the multiple flows, the bottlenecks of the multiple flows are identified as a test result. The performance verification apparatus according to feature 1.

3. The benchmark calculation unit, From among the benchmark items that are candidates for bottlenecks in the aforementioned multiple flows, select the benchmark item with the highest likelihood calculated from the information on the correlation between the multiple flows and the load configuration. The performance verification apparatus according to feature 1.

4. The benchmark calculation unit, The lower limit of the benchmark result is estimated from the test results and the bottlenecks of the multiple flows. The performance verification apparatus according to feature 3.

5. The benchmark calculation unit, Having prior information used to estimate the load configuration, The process of identifying bottlenecks in the aforementioned multiple flows and updating the prior information based on the test results are used. The performance verification apparatus according to feature 3.

6. The benchmark calculation unit, Supplemental information regarding the aforementioned multiple flows is received from an external source, and bottlenecks in the aforementioned multiple flows are identified based on the test results and the supplemental information. The performance verification apparatus according to feature 2.

7. The benchmark calculation unit, Based on the results of the aforementioned benchmark, we estimate the performance of flows that can be adopted in the future. The performance verification apparatus according to feature 1.

8. The aforementioned test execution unit, Based on the calculated benchmark results, modify at least one of the multiple flows, collect again information on the correlation between the multiple flows including the modified flow, and collect again test results including the information on the correlation between the multiple flows. The performance verification apparatus according to feature 1.

Images