Test execution device, method, and program

Abstract

The goal is to provide technology that enables reducing the time required for testing while keeping costs down. [Solution] A test execution device for performing multiple tests on an application, comprising memory and a processor, wherein the memory stores management information that enables the identification of a resource configuration of a test environment capable of executing each test, the processor identifies a resource configuration of a test environment to be used to execute each of the tests based on the management information, aggregates the tests for each resource configuration, constructs a test environment for each of the resource configurations, and sequentially executes the aggregated tests in the test environment.

Description

[Technical Field]

[0001] The present disclosure relates to a technology for executing tests on applications. [Background Art]

[0002] A style of rapidly providing services has become established, in which applications are developed in a short period of time using OSS (Open Source Software) and commercial external libraries, and the applications are run on resources created on a public cloud. However, for an application running on a public cloud, if there is any update, including not only when the application itself is version-upgraded, but also when only a library approaching EOL (End Of Life), a managed service provided by the public cloud, or only the platform is version-upgraded, it is necessary to perform a regression test to confirm that the application operates without problems. Regression testing requires a large number of test items that need to be checked to ensure the quality of the application, and thus takes a long time. Even if a development environment that automatically executes building and testing of the application during midnight hours when application developers do not perform coding is used, it still takes time for the regression test to be completed.

[0003] Patent Document 1 discloses a technology for efficiently executing tests by generating a test execution script for each operating environment (Web browser / OS / terminal) and executing test code in parallel on a plurality of clients. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Publication No. 2018-139106 [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] However, in the method described in Patent Document 1, the application being tested is not parallelized. Therefore, when running tests that involve a period during which the application does not provide normal functionality, such as a test that restarts the database which is a component of the application, other tests running in parallel during that period may fail. It is thought that by leveraging the advantages of public clouds, which can dynamically create resources, and building a number of independent test environments and running tests in parallel in each test environment, it would be possible to run multiple tests in parallel without worrying about the impact on other tests, thereby reducing the time required to complete all tests. However, with pay-as-you-go public clouds, resource usage fees are incurred even during the time it takes to create resources, initialize the application and database, and then start the tests, which increases costs. One of the purposes of this disclosure is to provide a technology that enables reducing the time required for testing while controlling costs. [Means for solving the problem]

[0006] One aspect of this disclosure is the following test execution device: A test execution device that performs multiple tests on an application, comprising memory and a processor, wherein the memory stores management information that enables the identification of a resource configuration of a test environment capable of executing each test, the processor identifies a resource configuration of a test environment to be used to execute each of the tests based on the management information, aggregates the tests for each resource configuration, constructs a test environment for each of the resource configurations, and sequentially executes the tests aggregated for the resource configuration of the test environment in the test environment. [Effects of the Invention]

[0007] According to one aspect of this disclosure, it is possible to reduce the time required for testing while controlling costs. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows the configuration of the test execution device. [Figure 2] The hardware configuration of the test execution device is shown in block diagram. [Figure 3] This diagram illustrates an example of a test environment. [Figure 4] This figure shows an example of a test item table. [Figure 5] This figure shows an example of the configuration of a test item / test type mapping table. [Figure 6] This figure shows an example of the configuration of the resource and configuration change table. [Figure 7] This figure shows an example of the configuration of a test item / resource mapping table. [Figure 8] The flow of the process for determining the test environment is shown in the flowchart. [Figure 9] This figure shows an example of the structure of an IaC storage table. [Figure 10] This figure shows an example of a file to be read during IaC modification processing. [Figure 11] This flowchart shows the detailed flow of the IaC change process. [Figure 12] This figure shows an example of the structure of a test execution table. [Figure 13] This flowchart shows the detailed flow of the test environment setup process. [Figure 14] This flowchart shows the detailed flow of the test execution process. [Figure 15] This figure shows an example of the test execution status screen displayed by the test execution status presentation process. [Figure 16] This block diagram shows the configuration of the test execution device. [Figure 17] This figure shows an example of a test item table in Example 2. [Figure 18] The flow of the process for determining the test environment is shown in the flowchart. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments of the present invention are described below with reference to the drawings.

[0010] The test execution apparatus is an apparatus that executes a plurality of tests on an application. Test execution is required to reduce the time required for testing while suppressing costs. FIG. 1 is a block diagram showing the configuration of the test execution apparatus.

[0011] The test execution apparatus 10 includes a processor 11, a main memory 12, and a storage device 13. Various types of management information are stored in the storage device 13. The functions of the test execution apparatus 10 are realized when the processor 11 executes software programs for various processes on the main memory 12 based on the management information on the storage device 13.

[0012] The management information is information that enables specifying the resource configuration of a test environment in which each test can be executed. Based on the management information, the processor 11 specifies the resource configuration of the test environment used to execute each of the plurality of tests. The resource configuration is defined by the conditions required for each resource to execute a test and check the items to be verified in the test. Here, as an example, it is assumed that a test environment is constructed on a public cloud, and the processor 11 constructs the test environment using resources provided by the public cloud. Further, the processor 11 aggregates tests for each specified resource configuration, constructs a test environment for each resource configuration, and sequentially executes the tests aggregated according to the resource configuration of the test environment in the corresponding test environment. Then, when all tests aggregated for the resource configuration of each test environment are completed, the processor 11 deletes the resources of the test environment on the public cloud. Accordingly, since tests are aggregated for each resource configuration of the test environment, each test environment is constructed, and tests are executed for each test environment, it is possible to reduce the time required for testing and suppress costs.

[0013] In the present embodiment, the management information may further be information that enables specifying the execution time, which is the time required to execute each test. Further, if the sum of the test execution times for each test environment of the tests aggregated for each resource configuration exceeds a predetermined threshold, the processor 11 may divide the tests aggregated for the resource configuration into a plurality of groups, construct a plurality of test environments with the resource configuration, and execute the tests divided into the plurality of test environments. According to this configuration, since the test environments are constructed and the tests are divided for execution such that the execution time for each test environment does not exceed the threshold, it is possible to reduce the time required for testing to the required time and suppress costs. Hereinafter, more specific examples of the present embodiment will be described. [Example]

[0014] The test execution device according to Example 1 has the configuration shown in Figure 1. Referring again to Figure 1, the software program running on the main memory 12 includes software programs for executing the test environment determination process 201, the IaC change process 202, the test environment construction process 203, the test execution process 204, and the test execution status presentation process 205, respectively.

[0015] The test environment determination process 201 identifies the test type for each test item, identifies the resource configuration for each test type, and determines the test environment to be prepared based on the identified resource configuration. The IaC change process 202 is the process of changing the IaC (Infrastructure as Code) information according to each determined test environment. IaC is a method of defining the test environment in code and building the test environment based on the files defined in that code, and it has become widespread for the efficient development of applications.

[0016] An example of IaC is the IaC service provided by AWS (Amazon Web Services). AWS and Amazon are registered trademarks. The original IaC information is information that shows the actual execution environment of the application in code. Not all tests require the same configuration as the actual execution environment. Depending on the test, the test objective can be achieved by running and verifying in a partially degraded test environment. In that case, the test should be run using a test environment with reduced or abridged resources from the original execution environment. The modified IaC information will show the modified test environment. Test environment construction process 203 is the process of building the test environment according to the modified IaC information. Test execution process 204 is the process of running the test in the built test environment. Test execution status presentation process 205 is the process of presenting the status of the test execution to the user.

[0017] The management information on the storage device 13 includes the test item table 211, the test item / test type mapping table 212, the resource / configuration change table 213, the test item / resource mapping table 214, the IaC save table 215, the test execution table 216, and the IaC file storage area 217.

[0018] The test item table 211 stores information about the test items for multiple tests that should be performed on the application. The test item / test type mapping table 212 stores information showing the correspondence between the test items and test types for each test.

[0019] The Resource and Configuration Change Table 213 stores information indicating the performance and configuration of each resource for each test environment of each test type. The performance and configuration of each resource are indicated by the changes from the original execution environment.

[0020] The test item / resource mapping table 214 stores information indicating the performance and configuration of each resource in the test environment for each test item of each test, based on the changes from the original execution environment. The test execution table 216 stores information for each test that indicates the status of the test and the test environment.

[0021] The IaC storage table 215 stores information indicating the correspondence between the storage location of the IaC information representing the original execution environment and the storage location of the IaC information representing the current test environment (the modified test environment) for each test environment. The IaC file storage area 217 stores files containing IaC information that represent the resource configuration of each test environment. Figure 2 is a block diagram showing the hardware configuration of the test execution device.

[0022] The test execution device 10 is a computer that includes a processor 11, main memory 12, and storage device 13, as well as a communication device 14, an input device 15, and a display device 16, all connected via a bus 17. The communication device 14 is for connecting to the application under test, the test client that executes the test, and the public cloud that constructs the test environment via a network. The input device 15 is for inputting information into tables set in the storage device 13. The display device 16 displays and outputs information such as the execution results from the processor 11.

[0023] Figure 3 is a diagram illustrating an example of a test environment. As mentioned above, the test environment is built on a public cloud. When running the test, the application 31 to be tested runs within the test environment on the public cloud.

[0024] Application 31 is broadly divided into a service portion 32 and a database portion 33. In the service portion 32, computing functions are realized by instances or containers 34 that perform processing. Furthermore, to ensure high availability, multiple instances or containers 34 that perform the same processing operate internally in a redundant configuration, so that even if one of them fails, the provision of computing services continues. Similarly, in the database portion 33, database functions are realized by a database 35 that stores data in a predetermined data structure. Furthermore, to ensure high availability, multiple databases 35 that store the same data operate internally in a redundant configuration, so that even if one fails, the provision of database functions continues.

[0025] The test client 36 may reside on a public cloud or within the network of an organization using a public cloud. The test client 36 sends a request message to an IP address called the representative address of the service portion 32 of the application 31, or to an FQDN (Fully Qualified Domain Name) resolved by DNS (Domain Name System). The instance or container 34 that receives the request message within the service portion 32 sends a query to the database portion 33 as needed, requesting an operation on the database 35. The database portion 33 also uses a representative address, so the service portion 32 does not need to be aware of which database 35 receives the query. The following describes the method for executing a test using the test execution device 10 configured as described above.

[0026] The test execution method includes a test environment determination process 201, an IaC change process 202, a test environment construction process 203, a test execution process 204, and a test execution status presentation process 205.

[0027] <Test environment determination process 201>

[0028] This section describes the management information used in the test environment determination process 201: the test item table 211, the test item / test type mapping table 212, the resource / configuration change table 213, and the test item / resource mapping table 214. Figure 4 shows an example of a test item table.

[0029] The test item table 211 consists of a test ID 41 that stores a unique value to identify the test item, an API 42 that stores the API (Application Programming Interface) of the request message sent from the test client 36 to the service part 32, a parameter 43 that stores the parameter part of the request message, a behavior 44 that stores the expected response from the application 31, a target resource 45 that stores the function that will be affected if some kind of failure is to be caused in the application 31 for the purpose of testing, and a situation 46 that stores the status of the failure.

[0030] Using the row with Test ID 31 and a value of 1, this test item involves sending a request message to the auth API, which indicates authentication, with parameters consisting of the value A stored in the key 'user' and the value 'aa' stored in the key 'password', while the database is restarting, which takes 60 seconds. The test expects the application to respond with the error message "DB access error". The contents of Test Item Table 211 are created in advance by the developer or tester during application development. Figure 5 shows an example of the configuration of a test item / test type mapping table.

[0031] The test item / test type mapping table 212 consists of a status 51 that stores a value corresponding to the status 46 set when the test is executed in the test item table 211 for each test, a test type 52 that stores the test type corresponding to the test, and a test type (large) 53 that further groups the test type of the test into a larger type.

[0032] Using the first and second lines, it is shown that both DB restart and DB failover are classified under the same DB reliability test type, and are classified under reliability as a major test type. The test item / test type mapping table 212 is created by the person in charge in advance of testing. The test item / test type mapping table 212 may be created for each application being developed, or it may be created and managed as a table common to multiple applications within the organization. Figure 6 shows an example of the configuration of the resource and configuration change table.

[0033] The Resource Configuration Change Table 213 contains information indicating how the test environment with the modified resource configuration from the original operating environment can be used, depending on the test type. The Resource Configuration Change Table 213 consists of a Resource Type 61 which stores the type of resource, a Test Type 62 which stores the test type, a Redundancy Configuration 63 which stores whether the resource's redundancy configuration can be changed, a Performance 64 which stores whether the resource's performance can be changed, and a Parallelism 65 which stores whether the test client can be used for multiple test items.

[0034] Here, resources include a database that provides database functionality and a service that provides computing functionality. Database performance is, for example, storage capacity and read / write speed. Service performance is, for example, computing clock speed and number of threads. In the original operating environment, both the service providing computing functionality and the database providing database functionality are assumed to have a redundant, dual configuration. Therefore, the notation "as is" for redundant configuration in the table indicates that the same redundant, dual configuration as the original operating environment will be adopted. Also, the notation "single" for redundant configuration in the table indicates that it is possible to change from the original redundant configuration to a single redundant configuration. Furthermore, the notation "as is" for performance in the table indicates that the same performance as the original operating environment will be adopted. Also, the notation "low" for performance in the table indicates that it is possible to reduce performance compared to the original operating environment.

[0035] Using the first three lines to explain, the first three lines store information about a test of type DB reliability. The first line indicates that the DB's redundant configuration remains the same as the original operating environment, and its performance also remains the same as the original operating environment. The second line indicates that the service's redundant configuration can be changed to single, and its performance can be reduced. The third line indicates that the test client's performance can be reduced, and the test client can be used in conjunction with other test environments. The Resource / Configuration Change Table 213 is created in advance by the person in charge prior to the test. The Resource / Configuration Change Table 213 may be created for each application being developed, or it may be created and managed as a common table for multiple applications within an organization. Figure 7 shows an example of the configuration of a test item / resource mapping table.

[0036] The test item / resource mapping table 214 consists of a test ID 71 which stores the value corresponding to the test ID 41 in the test item table 211, a client performance 72 which stores a value indicating the performance of the test client, a client co-use 73 which stores whether the test client is used in conjunction with other test environments, a service performance 74 which stores a value indicating the performance of the service, a service redundancy configuration 75 which stores a value indicating the redundancy configuration of the service, a database performance 76 which stores a value indicating the performance of the database, a database redundancy configuration 77 which stores a value indicating the redundancy configuration of the database, and a test environment 78 which stores a value that uniquely identifies the test environment.

[0037] The test item / resource mapping table 214 is created based on the contents of the test item table 211, the test item / test type mapping table 212, and the resource / configuration change table 213. Next, we will explain the test environment determination process 201. Figure 8 is a flowchart showing the flow of the test environment determination process.

[0038] The test environment determination process 201 is implemented by the processor 11 executing a software program, and can be said to be the main operating process. Therefore, the following explanation will focus on the test environment determination process 201.

[0039] Here, the initial state of the test item / resource mapping table 214 is empty. The resource configuration identification process 201 first retrieves a row of test items from the test item table 211 (S801). Next, the resource configuration identification process 201 refers to the test item / test type mapping table 212 and selects a row in which the value of status 51 matches the combined value of the target resource 45 and status 46 of the test item retrieved in step S801 (S802).

[0040] Next, the resource configuration identification process 201 retrieves the values ​​for resource type 61, redundant configuration 62, performance 64, and parallel usability 65 from the row in the resource / configuration change table 213 that matches the value of test type 52 of the row selected in step S802, and stores these internally along with the value of test ID 41 from the row in the test item table 211 retrieved in step S801 (S803).

[0041] Then, the resource configuration identification process 201 inserts a new row into the test item / resource mapping table 214 based on the table information obtained in steps S801, S802, and S803 (S804).

[0042] At that time, as values ​​for each column of the new row, first, the value of Test ID 41 from the Test Item Table 211 retrieved in step S801 is copied to Test ID 71. Here, in the Resource / Configuration Change Table 213, there are rows with a specific value for Test Type 62, corresponding to the number of resource types. In this embodiment, there are three types: "DB", "Service", and "Test Client". Therefore, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "DB", the value of Redundancy Configuration 63 is copied to the DB Redundancy Configuration 77 of the new row, and the value of Performance 64 is copied to the DB Performance 76 of the new row. Subsequently, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "Service", the value of Redundancy Configuration 63 is copied to the Service Redundancy Configuration 75 of the new row, and the value of Performance 64 is copied to the Service Performance 74 of the new row. Then, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "Test Client", the value of Performance 64 is copied to Client Performance 72 in the new row, and the value of Co-op 65 is copied to Client Co-op 73 in the new row.

[0043] Further, for the test environment 78 of a new row, if there is an existing row already inserted into the test item-resource mapping table 214 whose values of client performance 72, client combination 73, service performance 74, service redundant configuration 75, DB performance 76, and DB redundant configuration 77 completely match those of the new row, the value of the test environment 78 of said existing row is copied. If there is no existing row whose values of client performance 72, client combination 73, service performance 74, service redundant configuration 75, DB performance 76, and DB redundant configuration 77 completely match those of the new row, a value obtained by adding 1 to the maximum value among the values of the test environment 78 of existing rows already inserted into the test item-resource mapping table 214 is stored in the test environment 78 of the new row; if the test item-resource mapping table 214 is empty, that is, there is no existing row, 1 is stored in the test environment 78 of the new row. Then, the created new row is inserted at the end of the test item-resource mapping table 214.

[0044] Next, the test environment determination process 201 determines whether the row extracted from the test item table 211 in step S801 is the last row (S805), and if it is the last row, the resource configuration specification process 201 is ended. If it is not the last row, that is, if there are more rows, the test environment determination process 201 returns to step S801.

[0045] <IaC change processing 202> The IaC storage table 215, which is management information used in the test environment determination process 201, will be described. FIG. 9 is a diagram showing an example of the configuration of the IaC storage table.

[0046] The IaC storage table 215 includes a test environment 91 that stores a value corresponding to the test environment 78 of the test item-resource mapping table 214, an original IaC 92 that stores the location of the original IaC indicating the operating environment that the test environment is based on, and a post-change IaC 93 that stores the location of the IaC obtained by modifying the original IaC. A method of creating the IaC storage table 215 will be described later.

[0047] Figure 10 shows an example of a file read during IaC modification processing. Figure 10 shows an excerpt of IaC file 1001, written in YAML (YAML Ain't Markup Language) format used by CloudFormation, as an example of a file provided by the IaC service of AWS.

[0048] IaC file 1001 includes line 1002 indicating that the resource to be created is a service, line 1003 indicating the number of redundant instances if a redundant configuration is to be implemented, line 1004 indicating the number of CPU clocks to be allocated to the container if the service is a container, and line 1005 indicating the memory size. Regarding the database, it also includes lines 1006 and 1009 indicating that it is an instance with database functionality, lines 1007 and 1010 indicating an identifier that uniquely identifies the cluster when multiple instances with database functionality are configured redundantly as a single database functionality, and lines 1008 and 1011 indicating the instance performance. Instance performance can be indicated by directly specifying the CPU frequency and memory size, or by specifying a type of performance provided by the public cloud provider, as in line 1011. For the types of performance provided by the public cloud provider, the public cloud provider publishes the correspondence between the type and the specific CPU frequency and memory size in its official documentation.

[0049] Thus, for service functions, the value in row 1003 usually gives the number of services running. Similarly, for database functions, by identifying that they belong to the same cluster from the values ​​in rows 1007 and 1010, and then counting the number of instances in rows 1006 and 1009, the number of database instances running within the database function can be determined. Figure 11 is a flowchart showing the detailed flow of the IaC change process.

[0050] The IaC change process 202 is implemented by the processor 11 executing a software program, and it can be said that the IaC change process 202 is the main component of the operation. Therefore, the following explanation will focus on the IaC change process 202. IaC change process 202 is executed after the test environment determination process 201 is completed and the creation of the test item / resource mapping table 214 is complete.

[0051] First, the IaC change process 202 reads an IaC file from the IaC file storage area 217 on the storage device 13 of the test execution device 10 (S1101). Next, the IaC change process 202 retrieves one row from the test item / resource mapping table 214 (S1102). Subsequently, the IaC change process 202 determines whether the value of the test environment 78 in the row retrieved in step S1102 is the value of a test environment that has already been processed (S1103).

[0052] The IaC change process 202 proceeds to step S1108 if the value of the test environment 78 in the row retrieved in step S1102 is a value of a test environment that has already been processed; otherwise, it proceeds to step S1104.

[0053] In step S1104, the IaC modification process 202 modifies the configuration of the service portion of the IaC file 1001. Specifically, it refers to the value of service performance 74 in the row retrieved in step S1102 from the test item / resource mapping table 214. If the value is "low", it changes the performance values ​​1004 and 1005 of the service-related resources in the IaC file 1001 to a lower performance than the original performance. Furthermore, it refers to the value of service redundancy configuration 75. If the value is "single", it changes the numerical value 1003, which indicates the redundancy configuration of the service-related resource information in the IaC file 1001, to 1.

[0054] Next, in step S1105, the IaC modification process 202 modifies the database configuration in the same way that the service configuration was modified in step S1104. Specifically, it refers to the DB performance 76 value of the row retrieved in step S1102 from the test item / resource mapping table 214, and if the value is "low", it changes the values ​​of rows 1008 and 1011, which indicate the instance performance in the IaC file 1001, to lower values. Furthermore, it refers to the DB redundancy configuration 77 value, and if the value is "single", it reduces the number of rows related to the instance in the IaC file 1001 to just one (S1105). In this example, DBInstance2 is deleted, leaving only DBInstance1.

[0055] Next, the IaC modification process 202 stores the modified IaC file, which includes the configuration of the service portion modified in step S1104 and the configuration of the database portion modified in step S1105, in the IaC file storage area 217 and internally retains the storage location (S1106).

[0056] Furthermore, the IaC change process 202 updates the IaC storage table 215 to record the location of the modified IaC file (S1107). Specifically, it stores the location of the IaC file read in step S1101 in the original IaC 92, stores the location of the modified IaC file internally held in step S1101 in the modified IaC 93, and inserts a row into the IaC storage table 215 in which the value of the test environment 78 of the row retrieved from the test item / resource mapping table 214 in step S1102 is stored in the test environment 91, thereby internally holding the value of the test environment.

[0057] Next, the IaC change process 202 determines whether the row retrieved from the test item / resource mapping table 214 in step S1102 is the last row (S1108). If it is the last row, the IaC change process 202 terminates; otherwise, it returns to step S1102.

[0058] <Test environment setup process 203> This section describes the test execution table 216, which is management information used in the test environment setup process 203. Figure 12 shows an example of the configuration of a test execution table.

[0059] The test execution table 216 consists of a test environment 1201 which stores the value corresponding to the value of test environment 78 in the test item / resource mapping table 214, a test ID 1202 which stores the value corresponding to the value of test ID 41 in the test item table 211, a state 1203 which stores a value indicating the state of the test item indicated by the value of test ID 1202, a service 1204 which stores the representative address of the service function in the test environment indicated by the value of test environment 1201, a database 1205 which stores the representative address of the database function in the test environment indicated by the value of test 1201, and a result 1206 which stores the test result of the test item indicated by the value of test ID 1202. The test execution table 216 is updated sequentially in accordance with changes in the state of the test environment during the period from when the test environment is constructed by the test environment construction process 203 to when the tests are executed by the test execution process 206, and is initialized within the test execution process 206. Figure 13 is a flowchart showing the detailed flow of the test environment setup process.

[0060] The test environment setup process 203 is implemented by the processor 11 executing a software program, and it can be said that the test environment setup process 203 is the main operating component. Therefore, the following explanation will focus on the test environment setup process 203.

[0061] The test environment setup process 203 is called when application 31 becomes executable, for example, after building the latest source code of application 31 late at night, but before the test execution process 204 is executed.

[0062] First, the test environment construction process 203 retrieves one row from the IaC storage table 215 (S1301). Next, the test environment construction process 203 reads the IaC file from the location where the modified IaC 93 of the row retrieved in step S1301 is stored, executes the IaC, and starts building the test environment (S1302). Subsequently, the test environment construction process 203 retrieves a row from the test execution table 216 whose test environment 1201 has a value equal to the test environment 91 value of the row retrieved in step S1301, and sets the value of state 1203 to "building" (S1303). After that, the test environment construction process 203 determines whether it has finished retrieving all rows from the IaC storage table 215 (S1304). Since retrieving a row from the IaC storage table 215 means that test environment construction has started by executing the IaC for that row, it is determined here whether or not test environment construction has started for all rows.

[0063] If there are any rows that have not yet been imported, that is, if there are any rows for which the construction of the test environment has not yet begun, the test environment construction process 203 returns to step S1301 and imports the next row. If all rows have been imported, that is, if the construction of the test environment has begun for all rows, the test environment construction process 203 proceeds to step S1305.

[0064] In step S1305, the test environment setup process 203 retrieves one row from the test execution table 216. Then, the test environment setup process 203 obtains the progress of IaC corresponding to the value of the test environment 1201 of the row retrieved in step S1305, and updates the state 1203 of the corresponding row in the test execution table 216 to that progress (S1306). In addition to "Building," the progress statuses include "Completed," indicating that the environment setup is complete; "Rollback," indicating that a problem occurred during environment setup and rollback is in progress; and "Rollback Complete," indicating that the rollback of the environment setup is complete.

[0065] Furthermore, the test environment setup process 203 determines whether the value of the status 1203 updated in step S1306 is "completed" (S1307). If the value of the status 1203 is "completed", the test environment setup process 203 obtains the information necessary for access, such as the representative IP address and / or DNS name of the service and database created by IaC, and stores it in the service 1204 and database 1205 of the corresponding row in the test execution table 216 (S1308).

[0066] Subsequently, the test environment setup process 203 determines whether the value of status 1203 for all rows in the test execution table 216 is "Completed" or "Rollback Complete" (S1310). If the value of status 1203 for all rows is "Completed" or "Rollback Complete", the test environment setup process 203 terminates. If there are still rows where the value of status 1203 is "Building" or "Rolling Back", the test environment setup process 203 returns to step S1305.

[0067] <Test execution process 204> Figure 14 is a flowchart showing the detailed flow of the test execution process.

[0068] The test execution process 204 is a process implemented by the processor 11 executing a software program, and it can be said that the test execution process 204 is the main component of the operation. Therefore, the following explanation will focus on the test execution process 204.

[0069] The test execution process 204 first retrieves one row from the test execution table 216 (S1401). Next, the test execution process 204 compares the list of test items currently running for each test environment, which it has internally maintained, with the value of test environment 1201 in the row retrieved in step S1401, and determines whether there are any test items currently running in a test environment with the same resource configuration as test environment 1201 in the row retrieved in step S1401 (S1402). If there are test items currently running in the same test environment as test environment 1201 in the row retrieved in step S1401, the test execution process 204 proceeds to step S1410; otherwise, it proceeds to step S1403.

[0070] In step S1403, the test execution process 204 retrieves the service 1204 and database 1205 values ​​for the rows retrieved from the test execution table 216 in step S1401.

[0071] Next, the test execution process 204 selects a test client to be used for the test (S1404). Specifically, it extracts a row from the test item / resource mapping table 214 that has the same test ID 71 value as the test ID 1202 value of the row retrieved from the test execution table 216 in step S1401, and refers to the client performance 72 and client co-use 73 values ​​of that row. If the value of client co-use 73 is "not possible", a test client with performance corresponding to the value of client performance 72 is generated. The test client may be an instance, a container, or a service provided by a private cloud that temporarily reserves compute resources and runs programs. The location where the test client is generated does not matter as long as it can communicate with the test environment service, whether it is on a public cloud or not. If the value of client co-use 73 is "possible", an already generated test client is selected. However, if the number of test environments already allocated is greater than a predetermined number, a new test client may be generated.

[0072] Next, the test execution process 204 sets a specific status for the application, such as a failure, if necessary (S1405). Specifically, it extracts rows from the test item table 211 where the value of test ID 41 matches the value of test ID 1202 in the row extracted in step S1401, and refers to the target resource 45 and status 46 values ​​of that row. Then, using a service provided by a public cloud that intentionally generates failures, such as AWS FIS (Fault Injection Service), if the target resource 45 is "service", it specifies the resource using the value of service 1204 in the row extracted in step S1401, and if it is "DB", it specifies the resource using the value of database 1205 in the row extracted in step S1401, and sets the status by calling the API corresponding to status 46. Here, if the value of target resource 45 is "-" and the value of status 46 is "normal", it is a test when the application is operating normally, so no status is set.

[0073] Next, the test execution process 204 sends a request message (S1406) to the value of service 1204 in the row retrieved in step S1401, using the value of API 42 and the value of parameter 43 from the row of test item table 211 retrieved in step S1405. This causes the service of application 31 to execute the test and send a response message. If the response message received from the service matches behavior 44, the test is considered a success; otherwise, it is considered a failure.

[0074] Next, the test execution process 204 updates the test execution table 216 to set the value of the status 1203 of the row retrieved in step S1401 to "Completed" and the value of the result 1206 of that row to the test result of step S1406 (S1407). After that, the test execution process 204 determines in the test execution table 216 whether the status 1203 of all rows that have the same test environment 1201 value as the row retrieved in step S1401 is "Completed" (S1408). Then, if the value of the status 1203 of all those rows is "Completed", the test execution process 204 proceeds to step S1409, and if there are rows where the status 1203 is not yet "Completed", it proceeds to step S1410.

[0075] In step S1409, test execution process 204 deletes the test environment in which status 1203 was "Completed" for all rows. Specifically, if using AWS CloudFormation, you can simply execute the delete command again, specifying the stack name that was specified when the test environment was built.

[0076] Subsequently, the test execution process 204 determines whether the row to be processed, retrieved in step S1401, is the last row in the test execution table 216 (S1410). If the row to be processed is the last row in the test execution table 216, the test execution process 204 proceeds to step S1411. If the row to be processed is not the last row in the test execution table 216 and other rows remain, the test execution process 204 returns to step S1401. In step S1411, the test execution process 204 determines whether the status 1203 for all rows in the test execution table 216 is "completed" (S1411). If there are rows in the test execution table 216 whose status 1203 value is not "completed", the test execution process 204 resets the information of the read row and returns to step S1401. If the status 1203 value for all rows in the test execution table 216 is "completed", the test execution process 204 terminates processing.

[0077] <Test execution status display process 205>

[0078] The test execution status presentation process 205 is a process that displays the test execution status on the screen and presents it to the user. This test execution status presentation process 205 is implemented by the processor 11 executing a software program, and it can be said that the test execution status presentation process 205 is the main operating component. Therefore, the following explanation will focus on the test execution status presentation process 205 as the main component. Figure 15 shows an example of the test execution status screen displayed by the test execution status presentation process.

[0079] The test execution status screen 1501 has a string area 1502 that shows the execution status of the entire test, including multiple tests. If the value of status 1203 is "Completed" for all rows in the test execution table, the test execution status presentation process 205 displays "Completed" in the string area 1502. If there are any rows in the test execution table where the value of status 1203 is not "Completed", the test execution status presentation process 205 displays "Running" in the string area 1502.

[0080] Furthermore, the test execution status screen 1501 has an area 1503 that shows the status of each test. Area 1503 displays the status of each test in a table format. For each test, the table displays the test environment 1504, test ID 1505, status 1506, and result 1507. The test execution status presentation process 205 displays the value of test environment 1201 from the test execution table 216 in test environment 1504. The test execution status presentation process 205 also displays the value of test ID 1202 in test ID 1505. The test execution status presentation process 205 displays the value of status 1203 in status 1506. The test execution status presentation process 205 displays the value of result 1206 in result 1507.

[0081] Furthermore, the test execution status screen 1501 includes an area 1508 that shows the test execution status in the test environment selected by the user. Area 1508 displays the test execution status in a table format. The test execution status presentation process 205 displays the test execution status in area 1508 for the test environment corresponding to the value of test environment 1504 in the row selected by the user in area 1503. Area 1508 displays the test environment name, as well as the test ID 1509 and test execution status 1510 of the test executed in that test environment. For test ID 1509 and status 1510, the test execution status presentation process 205 displays the values ​​of test ID 1202 and status 1203 from the test execution table 216 for rows where the value of test environment 1201 matches the value of test environment 1504 in the row selected in area 1503.

[0082] Furthermore, the test execution status screen 1501 includes an area 1511 that displays detailed information about the test selected by the user. Area 1511 displays detailed test information in a table format. Specifically, it displays the test ID 1509, API 1513, parameter 1514, behavior 1515, target resource 1516, status 1517, and result 1518 as information about the test item in the row selected by the user in area 1508. The test execution status presentation process 205 retrieves the values ​​for API 42, parameter 43, behavior 44, target resource 45, and status 45 from the test item table 211 for the row where the value of test ID 41 matches the test ID 1509 of the row selected in area 1508, and displays them in area 1511 in the fields for API 1513, parameter 1514, behavior 1515, target resource 1516, and status 1517. Furthermore, the test execution status presentation process 205 displays the value of result 1206 in result 1518 in area 1511 for rows with a test ID that matches the value of test ID 1202 in the test execution table 216 with the value of test ID 1509 of the row selected in area 1508.

[0083] As described above, the test execution device 10 of this embodiment makes it possible to build a test environment with a resource configuration that suppresses the cost of using a public cloud in application development where it is necessary to perform tests on various items, and to shorten the overall time required for testing by executing tests in parallel in each test environment. [Examples]

[0084] Example 2 will be described using Figures 16 to 18.

[0085] Figure 16 is a block diagram showing the configuration of the test execution device in Example 2. In the test execution device of Example 2 shown in Figure 16, components and processes common to the test execution device of Example 1 shown in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0086] Referring to Figure 16, the test execution device 10 of Example 2 has a test environment determination process 1601 and a test item table 1611 instead of the test environment determination process 201 and test item table 211 in the test execution device 10 of Example 1.

[0087] <Test environment determination process 1601> Figure 17 shows an example of a test item table in Example 2.

[0088] Test item table 1611 is obtained by adding execution time 1701 to test item table 211 in Example 1. Execution time 1701 stores the time (in seconds) required to execute each test. The time required to execute each test may be the time expected from the content of the test, the maximum or average time taken when the same test was executed in the past, or a value predicted using machine learning based on data of the time taken to execute past tests. Figure 18 is a flowchart showing the flow of the test environment determination process in Example 2.

[0089] The test environment determination process 1601 is implemented by the processor 11 executing a software program, and it can be said that the test environment determination process 1601 is the main operating component. Therefore, the following explanation will focus on the test environment determination process 1601.

[0090] Referring to Figure 18, in Example 2, the test environment determination process 1601 performs the same steps S801, S802, and S803 as in Example 1. Then, in step S1601, a new row is inserted into the test item / resource mapping table 214 based on the information in the table obtained by steps S801, S802, and S803.

[0091] At that time, as values ​​for each column of the new row, first, the value of Test ID 41 from the Test Item Table 211 retrieved in step S801 is copied to Test ID 71. Here, in the Resource / Configuration Change Table 213, there are rows with a specific value for Test Type 62, corresponding to the number of resource types. In this embodiment, there are three types: "DB", "Service", and "Test Client". Therefore, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "DB", the value of Redundancy Configuration 63 is copied to the DB Redundancy Configuration 77 of the new row, and the value of Performance 64 is copied to the DB Performance 76 of the new row. Subsequently, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "Service", the value of Redundancy Configuration 63 is copied to the Service Redundancy Configuration 75 of the new row, and the value of Performance 64 is copied to the Service Performance 74 of the new row. Then, from the row in the Resource / Configuration Change Table 213 where the value of Resource Type 61 is "Test Client", the value of Performance 64 is copied to the Client Performance 72 of the new row, and the value of Co-op 65 is copied to the Client Co-op 73 of the new row. Up to this point, the process is the same as S804 in Example 1, but the process from here on is different from S804 in Example 1.

[0092] First, the test environment determination process 1601 checks whether there are any existing rows in the test item / resource mapping table 214 that perfectly match the values ​​of the new rows: client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, and DB redundancy configuration 77. If there are existing rows that perfectly match the values ​​of the new rows: client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, and DB redundancy configuration 77, the test environment determination process 1601 internally stores an accumulated value calculated by summing the values ​​stored in the execution time 1701 of the test item table 1611 for each test environment value. Then, the test environment determination process 1601 checks whether the total accumulated value, obtained by adding the execution time of the test item to be processed to this accumulated value, exceeds a predetermined threshold. For example, if you want to complete all tests in the middle of the night and perform tests from 1 AM to 6 AM, the threshold should be set to 5 hours (300 minutes, 18000 seconds).

[0093] If there is an existing row whose values ​​perfectly match those of the new row's client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, and DB redundancy configuration 77, and the total cumulative value does not exceed the threshold, then the test decision process 1601 copies the values ​​of the test environment 78 of the existing row whose values ​​perfectly match those of the new row's client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, and DB redundancy configuration 77 to the test environment 78 of the new row. If there is an existing row whose values ​​perfectly match those of the new row's client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, DB redundancy configuration 77, but the total cumulative value exceeds the threshold, or if there is no existing row whose values ​​perfectly match those of the new row's client performance 72, client co-use 73, service performance 74, service redundancy configuration 75, DB performance 76, DB redundancy configuration 77, the test determination process 1601 stores in the new row's test environment 78 a value obtained by adding 1 to the maximum value of the test environment 78, if there are any rows that have been inserted into the test item / resource mapping table 214 up to that point, or stores 1 in the new row's test environment 78 if the test item / resource mapping table 214 is empty, i.e., there are no existing rows.

[0094] As explained above, in this embodiment, by constructing test environments so that the execution time for each test environment does not exceed a threshold and executing the tests separately, it is possible to suppress the time required for testing within the required time while also suppressing costs.

[0095] The embodiments and examples described above are illustrative for explaining the present invention and are not intended to limit the scope of the invention thereto. Those skilled in the art can implement the present invention in various other forms without departing from the spirit of the invention.

[0096] Furthermore, the above embodiments and / or examples include the following; however, the contents included in these embodiments and / or examples are not limited to those described below.

[0097] (Item 1) A test execution device that performs multiple tests on an application has memory and a processor. The memory stores management information that enables the identification of the resource configuration of a test environment on which each test can be executed. The processor identifies the resource configuration of the test environment to be used to execute each of the tests based on the management information, aggregates the tests for each resource configuration, constructs a test environment for each resource configuration, and sequentially executes the aggregated tests in the test environment. This allows for the aggregation of tests for each resource configuration of the test environment, the construction of each test environment, the sequential execution of tests for each test environment, and the deletion of resources for a test environment after all tests have been completed, thereby reducing the time required for testing and suppressing costs.

[0098] (Item 2) In the test execution device described in item 1, the management information further includes information that enables the identification of the execution time, which is the time required to execute each of the tests. If the sum of the execution times of the tests aggregated for each resource configuration exceeds a predetermined threshold, the processor divides the tests aggregated for that resource configuration into multiple tests, constructs multiple test environments for that resource configuration, and executes the tests by dividing them among the multiple resource configurations. This makes it possible to reduce the time required for testing to the required time while suppressing costs by constructing test environments and executing the tests separately so that the execution time for each test environment does not exceed the threshold.

[0099] (Item 3) In the test execution device described in item 1, the application is software running on a public cloud, the test environment is built on the public cloud, and the processor deletes the resources of the test environment on the public cloud once all tests aggregated in the resource configuration of each test environment have been completed. As a result, since the resources of the test environment on the public cloud are deleted once all tests aggregated in the resource configuration of each test environment have been completed, the usage fees for public cloud resources can be kept low.

[0100] (Item 4) In the test execution device described in item 1, the memory stores a test item-test type mapping table defining the test type for each of the tests, and a resource-configuration change table defining the resource configuration required for each of the test types. The processor generates a test item-resource mapping table showing the resource configuration required for each of the tests based on the test item-test type mapping table and the resource-configuration change table, and identifies the resource configuration of the test environment used to execute each of the tests based on the test item-resource mapping table. This makes it possible to easily identify the resource configuration of the test environment by pre-defining the test type for each of the tests and the resource configuration required for each of the test types.

[0101] (Item 5) In the test execution device described in item 4, the resource and configuration change table defines, for each test type, the extent to which a test environment can be created with reduced or lowered resources from the original execution environment. This allows tests to be executed in a test environment with reduced or lowered resources from the original operating environment, thereby effectively reducing costs.

[0102] (Item 6) In the test execution device described in item 5, the resources include a service that provides computing functions and a database that provides database functions, and the test environment is determined by the required performance and redundant configuration for the service and the database, respectively. This allows tests to be performed in a test environment with possible reductions or mitigations of the service and / or database from the original operating environment, thereby effectively reducing costs.

[0103] (Item 7) In the test execution device described in item 6, the resources further include a client, and the test environment is determined by the required performance of the client and whether it can be used in conjunction with other tests. This allows tests to be performed in a test environment with a reduction or simplification of the client from the original operating environment, thereby effectively reducing costs.

[0104] (Item 8) In the test execution device described in item 1, the processor generates information for displaying the progress of test execution for each test environment. [Explanation of Symbols]

[0105] 10...Test execution device, 12...Main memory, 13...Storage device, 201,1601...Test environment determination process, 202...IaC change process, 203...Test environment construction process, 204...Test execution process, 205...Test execution status display process, 211,1611...Test item table, 212...Test item / test type mapping table, 213...Resource / configuration change table, 214...Test item / resource mapping table, 215...IaC save table, 216...Test execution table, 217...IaC file storage area

Claims

1. A test execution device that performs multiple tests on an application, It has memory and a processor, The memory stores management information that enables the identification of the resource configuration of the test environment in which each test can be executed, The aforementioned processor, Based on the aforementioned management information, the resource configuration of the test environment used to execute each of the aforementioned tests is identified. We aggregated tests for each resource configuration. A test environment is constructed for each of the aforementioned resource configurations. The tests, which are consolidated into the resource configuration of the aforementioned test environment, are executed sequentially within that test environment. Test execution device.

2. The aforementioned management information further includes information that makes it possible to identify the execution time, which is the time required to execute each of the aforementioned tests. The aforementioned processor, If the sum of the execution times of tests aggregated for each resource configuration exceeds a predetermined threshold, the tests aggregated for that resource configuration will be divided into multiple tests, multiple test environments for that resource configuration will be constructed, and the tests will be executed by dividing them among the multiple resource configurations. The test execution device according to claim 1.

3. The aforementioned application is software that runs on a public cloud. The aforementioned test environment is built on the aforementioned public cloud, The processor deletes the resources of the test environment on the public cloud once all tests aggregated in the resource configuration of each test environment have been completed. The test execution device according to claim 1.

4. The aforementioned memory is A test item / test type mapping table defines the test type for each of the aforementioned tests, A resource configuration change table defining the required resource configuration for the aforementioned test type is stored, The aforementioned processor, Based on the aforementioned test item / test type mapping table and the aforementioned resource / configuration change table, a test item / resource mapping table is generated that shows the resource configuration required for each of the tests. Based on the aforementioned test item / resource mapping table, the resource configuration of the test environment used to execute each of the aforementioned tests is identified. The test execution device according to claim 1.

5. The aforementioned resource and configuration change table defines, for each test type, the extent to which a test environment can be created with reduced or diminished resources compared to the original execution environment. The test execution device according to claim 4.

6. The aforementioned resources include a service that provides computing functions and a database that provides database functions. The test environment is determined by the required performance and redundancy configuration for the service and the database, respectively. The test execution device according to claim 5.

7. The aforementioned resources further include a client, The aforementioned test environment is determined by the required performance for the client and whether it can be used in conjunction with other tests. The test execution device according to claim 6.

8. The aforementioned processor, To generate information for displaying the progress of test execution for each of the aforementioned test environments, The test execution device according to claim 1.

9. A test execution method for running multiple tests on an application, Computers For each test, it stores management information that allows it to identify the resource configuration of the test environment in which the test can be executed. Based on the aforementioned management information, the resource configuration of the test environment used to execute each of the aforementioned tests is identified. We aggregated tests for each resource configuration. A test environment is constructed for each of the aforementioned resource configurations. The tests, which are consolidated into the resource configuration of the aforementioned test environment, are executed sequentially within that test environment. A test execution method to perform the task.

10. A test execution program for running multiple tests on an application, On the computer, For each test, it stores management information that allows it to identify the resource configuration of the test environment in which the test can be executed. Based on the aforementioned management information, the resource configuration of the test environment used to execute each of the aforementioned tests is identified. We aggregated tests for each resource configuration. A test environment is constructed for each of the aforementioned resource configurations. The tests, which are consolidated into the resource configuration of the aforementioned test environment, are executed sequentially within that test environment. A test execution program that performs the task.

Images