Foundation test execution support system and foundation test execution support method

Abstract

This makes it possible to achieve both a reduction in the number of test cases executed and a reduction in test execution time. [Solution] A system for supporting the execution of system infrastructure tests, which is performed by a computer having a processor and memory, wherein the processor determines the execution order of two or more test cases based on similarity information of system states between two test cases, specifically similarity information between the intermediate state of the system in the first test case and the prior state of the system in the second test case.

Description

Technical Field

[0001] The present invention relates to an infrastructure test execution support system and an infrastructure test execution support method.

Background Art

[0002] The importance of ensuring continuous security of IT systems is increasing. For example, border defense has become insufficient to ensure security against ransomware vulnerability attacks via internal networks. Against this background, to respond to continuously and unexpectedly discovered vulnerabilities in infrastructure software, it is important to sequentially update the infrastructure of IT systems. Along with this, the importance of repeated system tests has also increased, but since it is difficult to handle system tests manually alone, automation is required.

[0003] On the other hand, to ensure the reliability of IT systems, it is desirable to perform reliability tests (failure tests) with various failure patterns. In cloud environments, transparent reliability assurance by hardware is not provided, and users need to implement various redundancy measures. To reduce the burden of such measures, the importance of infrastructure failure tests is increasing. However, there is a problem that it takes time to execute infrastructure failure tests with various failure patterns. As a technique for solving such a problem, for example, there is Patent Document 1. In Patent Document 1, important test cases are extracted according to the program development status, test status, and the like.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 describes a method for extracting important test cases based on the program's development status and testing status. However, simply extracting and prioritizing important test cases may reduce the number of test cases executed, especially when there are constraints on total execution time. Reliability tests, in particular, tend to be time-consuming, making them prone to reducing the number of test cases executed. While it is possible to build and prepare a separate system for each test case and then execute the tests, this method has limitations such as resource limits and is costly. Therefore, there is a need for a technology that can both suppress the reduction in the number of test cases executed and shorten the test execution time.

[0006] Therefore, the present invention aims to provide a technology that can achieve both a reduction in the number of test cases executed and a reduction in test execution time. [Means for solving the problem]

[0007] The present invention relates to a foundation test execution support system, which is executed by a computer having a processor and memory, and is configured as a foundation test execution support system that assists in the execution of tests relating to the foundation of a system, wherein the processor is configured to specify the execution order of two or more test cases based on similarity information of the system state between two test cases, specifically between the intermediate state of the system in the first test case and the prior state of the system in the second test case. [Effects of the Invention]

[0008] According to the present invention, it is possible to suppress a decrease in the number of test cases executed and to shorten the test execution time simultaneously. [Brief explanation of the drawing]

[0009] [Figure 1]This figure shows an example of a system configuration to which a basic test execution support system, which is one embodiment of the present invention, is applied. [Figure 2] This figure shows an example of the hardware configuration of the management server. [Figure 3] This is a diagram illustrating an example of a test case. [Figure 4] Figure 3 illustrates the processing flow and system state when considering the test case shown. [Figure 5] Figure 4 illustrates the processing flow and system state, and shows the processes that are skipped. [Figure 6] This figure shows an example of the functional configuration of the management server. [Figure 7] This figure shows an example of a temporary system configuration information management table. [Figure 8A] This figure shows an example of a fault definition table. [Figure 8B] This figure shows an example of a failure behavior definition table. [Figure 8C] This figure shows an example of a definition table for the status acquisition setting method. [Figure 8D] This figure shows an example of a system requirements definition table. [Figure 9] This figure shows an example of a test case management table. [Figure 10] This figure shows an example of a test constraint management table. [Figure 11] This figure shows an example of a test case execution order management table. [Figure 12] This flowchart shows an example of the processing procedure performed by the management server in this embodiment. [Figure 13] Figure 12 is a flowchart showing an example of the processing procedure for the detailed processing of S1201. [Figure 14] Figure 12 is a flowchart showing an example of the processing procedure for the detailed processing of S1202. [Figure 15]This is a flowchart showing another example of the processing procedure of the detailed process of S1202 shown in FIG. 12. [Figure 16] This is a diagram showing an example of a pair temporary management table. [Figure 17] This is a flowchart showing an example of the processing procedure of the detailed process of S1203 shown in FIG. 12. [Figure 18] This is a diagram showing an example of a display screen output by the present system. [Mode for Carrying Out the Invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are illustrations for explaining the present invention, and omissions and simplifications are appropriately made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specifically limited, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. for easy understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0011] As examples of various types of information, descriptions may be given using expressions such as "table", "list", "queue", etc., but various types of information may be expressed by other data structures. For example, various types of information such as "XX table", "XX list", and "XX queue" may be referred to as "XX information". When describing identification information, expressions such as "identification information", "identifier", "name", "ID", and "number" are used, and these can be replaced with each other.

[0012] When there are a plurality of components having the same or similar functions, different subscripts may be added to the same reference numeral for description. Further, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0013] In the examples, the processes performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs the processing defined in the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include dedicated circuits that perform specific processing. Here, dedicated circuits include, for example, FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), CPLDs (Complex Programmable Logic Devices), etc.

[0014] The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in the embodiment, two or more programs may be implemented as one program, or one program may be implemented as two or more programs.

[0015] Figure 1 shows an example of the configuration of System 1 to which a basic test execution support system, which is one embodiment of the present invention, is applied. System 1 includes an IT system 101, an IT system management server 102, a source code repository 103, a management server 105, and a console 106, which are connected to each other via a network 104 so that they can communicate with one another.

[0016] IT system 101 is a server on which general IT systems such as web systems are built. IT system 101 is managed by IT system management server 102. IT system management server 102 is a server that manages IT system 101. IT system management server 102 manages the operating status of IT system 101, etc.

[0017] The source code repository 103 is a server that manages the source code and system configuration for running the IT system 101. The system configuration includes information that characterizes the system, such as the type and version of the operating system and applications, the clock speed of the CPU used as hardware, the size of the memory, and the capacity of the database. The management server 105 is a server that manages the entire system 1. The management server 105 controls the operation of the base test execution support system in this embodiment.

[0018] Console 106 is an administrative terminal for use by the administrator of System 1. Network 104 may use a general network such as the Internet.

[0019] Figure 2 shows an example of the hardware configuration of the management server 105. As shown in Figure 2, the management server 105 can be realized by a computer in which these components are connected to each other by internal communication lines such as a system bus. For example, the computer has a CPU 201, memory 202, auxiliary storage devices 203 such as ROM (Read Only Memory), a communication interface 204 such as a NIC (Network Interface Card) for connecting to a communication network, a media interface 205 for reading and writing information to portable storage media such as CDs (Compact Disks) and USB memory, and an input / output device 206 consisting of devices that accept various types of information input such as a keyboard, mouse, reader, and scanner, and a display that outputs various types of information that are input and used for processing.

[0020] Furthermore, the management server 105 is connected to the console 106 operated by the administrator and external storage devices 207 such as HDDs (Hard Disk Drives) via various networks, including network 104.

[0021] Various data stored in the management server 105 or used for processing can be accessed by the CPU 201 by reading from the auxiliary storage device 203 or the external storage device 207. Furthermore, each function of the management server 105 can be accessed by the CPU 201 by loading a predetermined program stored in the auxiliary storage device 203 or the external storage device 207 into memory 202 and executing it.

[0022] The aforementioned programs and data may be downloaded from a predetermined storage medium via the media interface 205, or from a network via the communication interface 204, loaded into memory 202, and executed by the CPU 201. Alternatively, they may be loaded directly into memory 202 from a predetermined storage medium via the media interface 205, or from a network via the communication interface 204, and executed by the CPU 201.

[0023] The following example illustrates a case where the management server 105 is configured on a single computer. However, all or part of these functions may be distributed across one or more computers, such as a cloud, and similar functions may be achieved by communicating with each other via a network. The same considerations apply to the other servers and systems shown in Figure 1.

[0024] Figure 3 is a diagram illustrating an example of a test case. Figure 3 illustrates three test cases assumed in the infrastructure test execution system run by the management server 105. In the test case, a fault is injected into one of the multiple zones (in this example, Zone 1 and Zone 2) that make up the IT system 101 in order to intentionally cause a failure in the IT system 101.

[0025] First, as Test Case 1, we consider injecting a failure into the primary DB in Zone 1, which contains the application (AP) and primary DB (Database), and failing over to the secondary DB in Zone 2. As Test Case 2, we consider injecting a failure into the primary DB in Zone 2, which contains the application and primary DB, and failing over to the secondary DB in Zone 1. As Test Case 3, we consider injecting a failure into the entire Zone 1, that is, both the application and the primary DB, and failing over to Zone 2.

[0026] Figure 4 is a diagram illustrating the processing flow and system state when the test case shown in Figure 3 is assumed. As shown in the upper part of Figure 4, first, for Zone 1 and Zone 2, the state of the applications and DB constituting each zone is initialized, the test case 1 is executed, fault injection and evaluation are performed, and a rollback is performed to return to the state before fault injection.

[0027] After rolling back Test Case 1, in order to run Test Case 2, the same process as in Test Case 1 is followed: initialization of the application and database state, fault injection and evaluation, and rollback. Similarly, for Test Case 3, the same process is followed: initialization of the application and database state, fault injection and evaluation, and rollback.

[0028] As shown in the lower part of Figure 4, the system state transitions as follows when the test is carried out with this processing flow. First, in the initial state of test case 1, when a failure is injected into the primary DB in Zone 1, the primary DB fails over to the secondary DB and takes over the data (S401). Then, in the state after the failure injection, the secondary DB that took over the data becomes the primary DB, and transactions and operations on the DB taken over are rolled back to the secondary DB, i.e., the primary DB in Zone 1 (S402). After the processing in S402, the state after the rollback is reached, and the primary DB in Zone 1 reflects the transactions and operations on the primary DB in Zone 2 that occurred in S402 (S403). The system state transitions similarly for test cases 2 and 3 that follow test case 1 (S404~S406, S407~S409).

[0029] When performing tests with multiple test cases, in this embodiment, as described below, for example, if the state after fault injection and the state before fault injection are the same between test cases, a shortcut process is implemented that skips the transition from consecutive rollbacks to initialization.

[0030] Figure 5 is a diagram illustrating the processes that are skipped in the processing flow and system state shown in Figure 4. In Figure 5, the rollback (S403) when moving from test case 1 to test case 2 and the rollback (S406) when moving from test case 2 to test case 3 are skipped, respectively, in the processing flow and system state shown in Figure 4. Therefore, in this embodiment, as shown in the processing flow, after the state initialization, fault injection, and evaluation of test case 1 are performed, a rollback is performed after the fault injection and evaluation of test case 2 and the fault injection and evaluation of test case 3 are performed.

[0031] Figure 6 shows an example of the functional configuration of the management server 105. In order to perform the processes described in Figure 5, the management server 105 has a temporary system configuration information management table 601 and various test definition tables 602. The temporary system configuration information management table 601 is obtained from the IT system management server 102. The various test definition tables 602 are assumed to be stored in advance.

[0032] Furthermore, the management server 105 has a test case generation unit 603. The test case generation unit 603 receives information from the system configuration information temporary management table 601, various test definition tables 602, and source code and system configuration information obtained from the source code repository 103 that describes the configuration for operating the IT system 101, and outputs and stores a test case management table 604.

[0033] Furthermore, the management server 105 includes a test case execution order identification unit 605 that identifies the execution order of test cases using a test constraint management table 606 stored on the server and generates a test case execution order management table 607, and a test case execution unit 608 that executes test cases using the test case execution order management table 607. The test case execution unit 608 executes test cases against the IT system 101 and feeds back the test results for the executed test cases to the test case management table 604.

[0034] Figure 7 shows an example of the temporary system configuration information management table 601. The temporary system configuration information management table 601 contains information indicating the configuration of the IT system 101 during the execution of processing by this system, and can be obtained from the IT system management server 102 or the source code repository 103. The temporary system configuration information management table 601 is also used when determining whether or not test cases need to be updated.

[0035] As shown in Figure 7, the system configuration information temporary management table 601 stores the following in association: resource ID 701 for identifying the resources owned by the IT system 101, resource type 702 indicating the type of resource identified by resource ID 701, resource group 703 indicating the group to which the resource identified by resource ID 701 belongs, and area 704 indicating the area where the resource identified by resource ID 701 is held. In Figure 7, for example, a resource with resource ID 701 "DB11" has resource type 702 "DB" (database), and the resource belongs to resource group 703 "DB Cluster 1" under area 704 "Zone1, Region1".

[0036] Figure 8A shows an example of a fault definition table 801. Figure 8B shows an example of a fault behavior definition table 802. Figure 8C shows an example of a status acquisition setting method definition table 803. Figure 8D shows an example of a system requirements definition table 804. These tables shown in Figures 8A to 8D constitute the various test definition tables 602 shown in Figure 6.

[0037] The failure definition table 801 is a table that defines resource failures in the IT system 101. As shown in Figure 8A, the failure definition table 801 stores a resource type 811 indicating the type of resource, a failure type 812 indicating the type of failure for that resource type, and a failure injection method 813 for that resource type. In Figure 8A, for example, it is shown that when injecting a failure of type "failover" into resource type "DB", the failure injection method "db_failover" is used.

[0038] The failure behavior definition table 802 is a table that defines the behavior of a resource into which a failure has been injected. As shown in Figure 8B, the failure behavior definition table 802 has resource type 821, which is the same as resource type 811 shown in Figure 8A, and failure type 822, which is the same as failure type 812. Corresponding to these items, the table stores the pre-state 823, which shows the state of the resource before failure injection, the failure injection target 824, which shows the resource that is a candidate for failure injection, and the post-state 825, which shows the state of the resource after failure injection. In Figure 8B, for example, when a failure of failure type "failover" is injected into resource type "DB" (the pattern in the first record), the pre-state is defined as DBa being "primary" and DBb being "secondary". When a failure is injected into DBa ("Y"), the post-state is defined as DBa becoming "secondary" and DBb becoming "primary". DBa corresponds to DB11 shown in Figure 8A, and DBb corresponds to DB12 shown in Figure 8A. This example assumes that there is one primary and one secondary database, but each can consist of multiple databases. For example, if there are four databases, two of them can be configured as primary databases and the other two as secondary databases, and the pre- and post-states and fault injection targets can be defined for each of them. This allows for the definition of each state for each database.

[0039] The status acquisition and setting method definition table 803 is a table that defines how to acquire and set the status of a resource. As shown in Figure 8C, the status acquisition and setting method definition table 803 stores the resource type 841, which indicates the type of resource whose status is to be acquired and set; the status type 842, which indicates the type of status of that resource; the acquisition method 843 for acquiring that type of status; and the setting method 844 for setting the acquired type of status. In Figure 8C, for example, when acquiring the status type "Role" for resource type "DB", it is shown that the acquisition method "get_db_role" is used. Also, when setting that status type, it is shown that the setting method "set_db_role" is used.

[0040] The system requirements definition table 804 is a table that defines the requirements when a failure occurs in the IT system 101. As shown in Figure 8D, the system requirements definition table 804 stores the requirement type 851, which indicates the type of requirement for the system, and the criteria 852 for that type of requirement in association with each other. In Figure 8D, for example, "t1" is defined as the recovery time requirement when a failure occurs in the IT system 101, and "t2" is defined as the response time requirement. In other words, when a failure occurs in the IT system 101, these requirements must be met.

[0041] Figure 9 shows an example of a test case management table 604. The test case management table 604 is a table for managing test cases for the IT system 101. As shown in Figure 9, the test case management table 604 stores the following information in association with each other: a test case ID 901 to identify the test case, a fault injection target 909 which is the resource to be injected with a fault, a fault type 910 which indicates the type of fault, a pre-fault state 903 which indicates the state of the system and resources before fault injection, a post-fault state 904 which indicates the state of the system and resources after fault injection, an execution content 905 which indicates the specific content of the test case, a history 906 which indicates the results of the test executed with the content shown in the execution content 905, an execution fee 907 which indicates the cost when the test was executed, and an execution time 908 when the test was executed. Here, as an example, the system (IT system 101) and DB (DB which constitutes the IT system 101) are stored as the state before and after the fault as the pre-fault state. Also, initialization, abnormal injection, and rollback are stored as examples of execution content. The states in which these actions are performed correspond to the initial state, the state after fault injection, and the state after rollback in the test case shown in Figure 5.

[0042] Figure 9 shows, for example, that test case ID "TestCase1" is a test that injects a failure that causes DB11 to fail over. The state before the failure is that IT system 101 is writing to DB11, which is running as the primary, and DB12, which is running as the secondary. The state after the failure is that IT system 101 is writable to DB11, which is running as the secondary, and DB12, which is running as the primary. Furthermore, this test shows that the commands to initialize "set_role(DB11, primary)system_start()", the command to inject an abnormality "db_failover(DB11)", and the command to rollback "db_failover(DB12)" are input. Currently, the number of successes and failures for this test is "0". If executed, it would cost "10 dollars", and the execution time would be "10 minutes" for the abnormal injection and "10 minutes" for the rollback. The success or failure of a test can be determined, for example, by whether or not the executed process completed successfully.

[0043] Figure 10 shows an example of a test constraint management table 606. The test constraint management table 606 is a table that defines the conditions that constrain the execution of tests. As shown in Figure 10, the test constraint management table 606 stores a test constraint type 1001 that indicates the type of constraint condition, and a specific value 1002 of that type in association with it. In Figure 10, for example, when performing a test on IT system 101, the total execution cost is set to "V1" and the total execution time is set to "V2" as upper limits.

[0044] Figure 11 shows an example of the test case execution order management table 607. The test case execution order management table 607 stores a test case ID 1101, similar to the test case ID 901 shown in Figure 9, an execution order 1102 that defines the order in which the test cases are executed, and a skip content 1103 that defines which states of the system and resources to skip. In Figure 11, for example, the first test case to be executed is identified as "TestCase1," and the content to be skipped in this test case is "the entire rollback." The second test case to be executed is identified as "TestCase2," and the content to be skipped in this test case is "the entire initialization and the entire rollback." Furthermore, the third test case to be executed is identified as "TestCase3," and the content to be skipped in this test case is "the entire initialization." After these three test cases are executed, the system state transitions as shown in Figure 5.

[0045] Figure 12 is a flowchart showing an example of the processing procedure performed by the management server 105 in this embodiment. As shown in Figure 12, the management server 105's test case generation unit 603 identifies whether or not a test case needs to be updated, and if an update is necessary, it performs the test case generation and update of the test case management table 604 (S1201). The specific processing of S1201 will be described later using Figure 13.

[0046] Next, the test case execution order identification unit 605 identifies whether each test case should be executed and its execution order based on the test case management table 604, and saves this information in the test case execution order management table 607 (S1202). The specific processing in S1202 will be described later using Figures 14-16.

[0047] Next, the test case execution unit 608 executes the test cases based on the execution order of the test cases stored in the test case execution order management table 607, and reflects the execution results in the test case management table 604 (S1203). The specific processing of S1203 will be described later using Figure 17.

[0048] Figure 13 is a flowchart showing an example of the processing procedure for the detailed processing of S1201 shown in Figure 12. As shown in Figure 13, the test case generation unit 603 identifies whether or not a test case needs to be updated based on the information in the temporary system configuration information management table 601 (Figure 7), and determines whether or not it needs to be updated (S1301, S1302). The information in the temporary system configuration information management table 601 is obtained from the IT system management server 102 or the source code repository 103. If a record is stored in the temporary system configuration information management table 601, or if a new record has been added, the test case generation unit 603 identifies the record and determines that an update to the test case is necessary.

[0049] If the test case generation unit 603 determines that an update to the test cases is not necessary (S1302; No), it terminates this process. However, if it determines that an update to the test cases is necessary (S1302; Yes), it proceeds to S1303.

[0050] The test case generation unit 603 identifies the target for fault injection based on the system configuration information in the temporary system configuration information management table 601 (S1303). In Figure 7, for example, the test case generation unit 603 identifies resource "DB11", whose resource type 702 is "DB" (database) and which belongs to resource group 703 "DB Cluster 1" under area 704 "Zone1, Region1", as the target for fault injection. Note that although the case of fault injection on a DB basis is described here, it may also be done on an area basis.

[0051] The test case generation unit 603 generates patterns (test patterns) of injection failure, pre-failure state, and post-failure state based on the failure injection target identified in step S1303 and the failure behavior information in the failure definition table 801 (Figure 8A) and failure behavior definition table 802 (Figure 8B) (S1304). For example, for the resource "DB11" identified as the failure injection target using Figure 7, the pre-failure injection state, injection failure, and post-failure injection state corresponding to the resource type and failure type shown in Figures 8A and 8B are generated as a single test pattern and become one record in the test case management table 604 in Figure 9. The items in the tables shown in Figures 8A and 8B have already been explained and will be omitted here. For the pre-failure state and post-failure state of the system ("writing" and "writable"), values ​​corresponding to the respective states shall be set.

[0052] Furthermore, the test case generation unit 603 identifies the initialization and rollback methods for each test pattern based on the pre-failure and post-failure states, and creates a test case (S1305). For example, the test case generation unit 603 reads the failure definition table 801 in Figure 8A, the failure behavior definition table 802 in Figure 8B, and the state acquisition setting method definition table 803 in Figure 8C, and writes the execution content corresponding to the test pattern generated in S1304. Initialization 917 can be set from the setting method 844 in Figure 8C, and abnormal injection 918 and rollback 919 can be set from the failure injection method in Figure 8A.

[0053] The test case generation unit 603 updates the test case management table 604 based on the information in the test case management table 604, reusing the execution fee and execution time information based on its similarity to existing test cases (S1306). For example, in S1305, the test case generation unit 603 determines whether the pre-failure state 903 and post-failure state 904 of the test case management table 604 are similar to a certain extent among past records executed as test cases. This determination may also include the execution content 905. If they are similar to a certain extent, the test case generation unit 603 determines that there is similarity between the two and sets the values ​​of these items (for example, execution fee 907, execution time 908) of the test case already written as a record as items of the newly written record. When the process in Figure 13 is completed, the test case management table 604 shown in Figure 9 is generated.

[0054] Figure 14 is a flowchart showing an example of the processing procedure for the detailed processing of S1202 shown in Figure 12. As shown in Figure 14, the test case execution order identification unit 605 selects one test case based on the information in the test case management table 604 (Figure 9) generated in S1201, and sets it as the N=1 test case (S1401). For example, the test case execution order identification unit 605 selects the record for "TestCase1" from the test case management table 604 shown in Figure 9.

[0055] Next, the test case execution order determination unit 605 selects a test case from among the undetermined test cases whose pre-failure state (pre-failure state) is close to the post-failure state (post-failure state) of the Nth test case, and makes it the N+1th test case (S1402). The test case execution order determination unit 605 selects, for example, a record other than the "TestCase1" record selected in S1401, whose pre-failure state 903 is close to the post-failure state 904 of the "TestCase1" record, and makes it the next test case in the sequence (for example, N=2). The above close test case is, for example, a test case in which the post-failure state 904 of the Nth test case and the pre-failure state 903 of the N+1th test case match. The two do not necessarily have to match; for example, they may be considered to match if they contain at least a predetermined number of matching roles or startup states, or if they are semantically similar to a certain extent. In this case, the test case execution order determination unit 605 may determine the execution order of the test cases in order of decreasing degree of agreement.

[0056] The test case execution order determination unit 605 determines whether the execution order of all test cases has been determined (S1403). If it determines that the execution order of all test cases has not been determined (S1403; No), it returns to S1402 and continues determining the execution order.

[0057] On the other hand, if the test case execution order identification unit 605 determines that the execution order of all test cases has been identified (S1403; Yes), it reflects the identified execution order in the test case execution order management table 607 shown in Figure 11 (S1404). For example, the test case execution order identification unit 605 writes the execution order of "TestCase1" as "1" and the execution order of "TestCase2" as "2" to the test case execution order management table 607. Furthermore, the test case execution order identification unit 605 identifies the execution content 905 (for example, initialization 917 and rollback 919) of the test case selected as the closest test case in S1402 as skippable processes, and writes these skippable processes to the skip content 1103 of the test case execution order management table 607 shown in Figure 11. When the process in S1404 is completed, the detailed processing in S1202 is completed. According to the processing procedure shown in Figure 14, the execution order of test cases can be determined without placing a processing load on the system compared to the processing procedure shown in Figure 15, which will be described later.

[0058] Figure 15 is a flowchart showing another example of the processing procedure for the detailed processing of S1202 shown in Figure 12. As shown in Figure 15, the test case execution order identification unit 605 identifies the execution priority of test cases based on the past execution history of test cases stored in the test case management table 604 shown in Figure 9, and in the following steps S1502 to S1505, it excludes test cases with low priority (S1501). For example, the test case execution order identification unit 605 determines that test cases with a history of 906 where the number of successes 920 is higher than a predetermined threshold have low priority because there is less need to execute them again, and excludes such test cases. Here, the number of successes 920 was used for exclusion, but if the execution fee 907 is higher than a predetermined threshold, or if the execution time 908 (abnormal injection 922, rollback 923) is longer than a predetermined threshold (i.e., takes a long time), the above priority may be determined to be low, and the test cases may be excluded. Furthermore, the remaining test cases may be sorted by the lowest number of successes (920), the lowest execution cost (907), or the shortest execution time (908) before proceeding with further processing. In other words, the success rate, execution cost, and execution time may be reflected in the test case priority before processing.

[0059] For example, as shown in Figure 9, the test case management table 604 manages the success or failure history of past test executions (success 920, failure 921 in Figure 9). Therefore, the test case execution order identification unit 605 may reflect the above success or failure history stored as the result of past test executions in the priority of the test cases, and perform the sorting after excluding test cases where the number of successes is higher than a predetermined threshold and the number of failures does not meet the predetermined threshold.

[0060] Next, the test case execution order identification unit 605, for each ordered pair of two test cases selected from all test cases (execution order when two test cases are executed consecutively), identifies whether there are any processes that can be skipped when the two test cases are executed consecutively, taking into account the similarity between the state before rollback (post-failure state) and during rollback processing in the first test case to be executed and the initial state (pre-failure state) in the second test case, and reflects this in the cost information of the ordered pair (S1502).

[0061] For example, consider the case where the test case execution order determination unit 605 selects the test case "TestCase3" shown in Figure 9 as the test case to be executed first. The test case execution order determination unit 605 determines that the two are similar if the post-failure state 904, which is the state before rollback, and the state during the rollback process, where the "db_failover(DB12)" of the rollback 919 of the execution content 905 is executed (the state in which ap_start(AP) is not executed), match the pre-failure state 903, which is the initial state of the next test case, "TestCase4" (not shown). The test case execution order determination unit 605 then determines that the two are similar if they match. The "ap_start(AP)" process, which is performed in the above matching state, is a process that can be skipped, and it associates the two test cases, "TestCase3" and "TestCase4", and writes them to the pair temporary management table 1600 for temporarily storing the two test cases. At this time, the test case execution order identification unit 605 writes a value corresponding to the degree of similarity as cost information to the pair temporary management table 1600. Here, if the post-failure state 904 before rollback and the state during the rollback process match the pre-failure state 903 of the next test case, the value corresponding to the degree of similarity between the two is set to "0," and the cost information decreases as the similarity increases. Thus, the cost information can be said to be an indicator of the similarity of the system states between two test cases.

[0062] In S1502, cost information was determined based on the degree of state agreement. However, cost information may also be determined by the execution time reduced when two test cases are executed consecutively, a value that takes into account whether some processing can be skipped when executed consecutively (for example, the number of processes to skip), or the execution time when skippable processing is skipped during consecutive execution.

[0063] Figure 16 shows an example of a pair temporary management table 1600. As shown in Figure 16, the pair temporary management table 1600 stores the following in association: test case 1601, which is the test case to be executed first as described above; next test case 1602, which is the test case to be executed next as described above; and cost 1603, which is the cost information described above.

[0064] Here, similarity was explained using the above-mentioned matching state as an example. However, as with determining the execution order in S1402, it is not always necessary for them to match. For example, if at least a part of them match, or if they are semantically similar to a certain extent, they may be considered to match. In this case, the test case execution order identification unit 605 may determine the cost information in order of decreasing degree of matching.

[0065] Returning to Figure 15, the test case execution order determination unit 605 determines the execution order of each target test case based on the cost information of each ordered pair of test cases (S1503). For example, the test case execution order determination unit 605 determines the execution order such that pairs with smaller cost information values ​​associated with pairs (subsets) of test cases written in the pair temporary management table 1600 shown in Figure 16 are executed earlier. In Figure 16, if the cost information "C11" associated with the pair "TestCase1" and "TestCase2" and the cost information "C22" associated with the pair "TestCase2" and "TestCase3" are smaller in this order than the cost information values ​​associated with other pairs of test cases, the test cases will be executed in the order "TestCase1", "TestCase2", and "TestCase3".

[0066] In this case, the test case execution order identification unit 605 may identify a subset of test cases (the lower-cost subset in Figure 16) from the defined test case execution order that does not exceed the upper limit of the total execution time in the test constraint management table 606 shown in Figure 10, and determine the execution order of the test cases in order of lowest total execution time. Similarly, the test case execution order identification unit 605 may identify a subset of test cases (the lower-cost subset in Figure 16) from the defined test case execution order that does not exceed the upper limit of the total execution cost in the test constraint management table 606 shown in Figure 10, and determine the execution order of the test cases in order of lowest total execution cost.

[0067] Next, the test case execution order identification unit 605 identifies skippable processes in each test case based on the identified test case execution order and reflects them in the test case execution order management table 607 (S1504). The skippable processes identified above are the processes identified in S1502. For the test cases whose execution order was identified in S1503, the test case execution order identification unit 605 writes the skippable processes identified in S1502 to the skip contents 1103 of the test case execution order management table 607 shown in Figure 11. For example, the test case execution order identification unit 605 writes the "ap_start(AP)" process (or rollback 919) of "TestCase3", which was identified as a skippable process in S1502, to the skip contents 1103.

[0068] Furthermore, the test case execution order identification unit 605 identifies test cases that satisfy the constraints in the test constraint management table 606 based on the test case execution order and skippable processes identified up to S1504, and reflects them in the test case execution order management table 607 (S1505).

[0069] For example, the test case execution order identification unit 605 refers to the execution order of the test cases identified in S1503, the skippable processes identified in S1504, and the test constraint management table 606 shown in Figure 10, and limits the scope of test cases to be executed so that the number of test cases increases while satisfying the conditions that the total execution cost is less than or equal to "V1" and the total execution time is less than or equal to "V2". The result is then written to and updated in the test case execution order management table 607. The determination of whether the sum of the processes performed in each test case is less than or equal to the above total execution cost and total execution time can be obtained from the execution cost 907 and execution time 908 in the test case management table 604 shown in Figure 9. The total execution cost and total execution time stored in the test constraint management table 606 can be determined based on the requirements for conducting the test. When the processing in S1505 is completed, the detailed processing in other examples of S1202 is completed.

[0070] The processes in Figures 14 and 15 both aim to reduce the total execution time of the tests by identifying the execution order that maximizes the time savings achieved through the shortcut process (skipping processes). In Figure 15, this is equivalent to constructing a directed graph where test cases are considered nodes, and consecutive executions between test cases are considered weighted edges. In other words, the shortest path passing through all nodes on the directed graph is determined to be the optimal execution order, i.e., the path that maximizes the time savings. For example, in Figure 15, unnecessary test cases are removed from the test execution history of the test cases, and the identity of the "post-state" and "pre-state" between test cases is determined. If identity is found, the weight of the edge is reduced (for example, by the expected time savings). Then, an approximate solution of the shortest path passing through all test cases is identified using known techniques, and the feasibility of a shortcut is determined at each edge along the path. Then, if it is determined that shortcuts are possible, based on the "order of nodes passed = order of test case execution," each test case is executed while skipping unnecessary processes, the execution time is measured, and the execution results are reflected in the history (for example, the number of test execution successes / failures, the execution time of each process).

[0071] Figure 17 is a flowchart showing an example of the processing procedure for the detailed processing of S1203 shown in Figure 12. As shown in Figure 17, the test case execution unit 608 executes each test case in the test case execution order management table 607 that has been updated up to S1202, measures the execution time, measures or calculates the execution fee, and obtains the execution results. If there are any skippable processes in the execution of the above test cases, they are skipped (S1701). For example, the test case execution unit 608 executes the tests defined in the test cases "TestCase1", "TestCase2", and "TestCase3" in the order defined in execution order 1102, and obtains the execution time, execution fee, and execution results. The execution time, execution fee, and execution results can be obtained from the log at the time of test execution, and the fee can be determined based on a predetermined fee.

[0072] Next, the test case execution unit 608 reflects the execution results, execution fee, and execution time of the processing in S1701 for each test case in the test case management table 604 (S1702). Once the processing in S1702 is completed, the detailed processing in S1203 is completed.

[0073] Figure 18 shows an example of a display screen output by this system. As shown in Figure 18, the display screen 1800 includes a test execution result 1801 showing the results of tests performed by the test case execution unit 608, and a test execution result (details) 1802 showing the details of the test execution results. The test execution result 1801 displays the test type, which indicates what type of test was performed, the number of successful tests, which indicates the number of tests that were successful, and the total number of successful and unsuccessful tests. Various types of tests, including the failure tests shown in this embodiment, can be displayed for the test type. The number of successful tests can be displayed based on the execution results obtained in S1701 shown in Figure 17, showing the total number of successful test cases among the executed test cases. The number of tests can be displayed as the total number of executed test cases.

[0074] Furthermore, the items displayed in the test execution results (details) 1802 can be read and displayed from each item in the test case management table 604 shown in Figure 9 and each item in the test case execution sequence management table 607 shown in Figure 11, as well as the execution results obtained in S1701 shown in Figure 17.

[0075] As explained above, this system makes it possible to achieve both a reduction in the number of test cases executed and a reduction in test execution time.

[0076] Specifically, as explained in S1502 of Figures 12 and 15, in a foundation test execution support system (e.g., management server 105) that is run by a computer (e.g., management server 105) having a processor and memory, the processor identifies the execution order of two or more test cases (e.g., execution order 1102 in Figure 11 based on the record in Figure 16) based on the similarity information between the system states of two test cases, specifically the intermediate state of the system in the first test case (e.g., the state after executing rollback 919 "db_failover(DB12)" in "TestCase3" in Figure 9 (a state where ap_start(AP) is not executed)) and the prior state of the system in the second test case (e.g., the pre-failure state 903 which is the initial state of "TestCase4" (not shown)) (e.g., judgment of whether these states are similar or not, S1502 in Figure 15). This allows us to determine the execution order of two test cases based on the similarity between their pre- and post-states.

[0077] Furthermore, as explained in each item in Figure 9, the processor makes the above identification based on the similarity information of the system state, which includes one or more of the following elements: (1) a state indicating whether the database possessed by the system is the primary or secondary configuration (role in the pre-failure and post-failure states in Figure 9), (2) a state indicating whether the system is writable or currently writing to it (systems 911 and 914 in the pre-failure and post-failure states in Figure 9), (3) a state indicating whether the database is running or not (startup state in the pre-failure and post-failure states in Figure 9), and (4) the number of databases. This makes it possible to identify the execution order of test cases in these specific system states.

[0078] Furthermore, as explained in S1502 of Figure 15, the processor determines an index indicating the similarity of the system state between the two test cases (for example, the cost information in S1502 of Figure 15) based on one of the following: the execution time reduced when the two test cases are executed consecutively, a value that takes into account whether some processing can be skipped when they are executed consecutively, or the execution time when skippable processing is skipped during consecutive execution. This allows the similarity of these test cases to be determined based on their execution time, reduction time, and the number of skips.

[0079] Furthermore, as explained in the items in Figures 15 and 9, the above-mentioned base test execution support system has a test case management table (Figure 9) for managing the time required for the rollback of the system's database (for example, the rollback of execution time 908 in Figure 9, 923). The processor makes the above identification based on the similarity information between the intermediate state of the system in the first test case relating to the failure test of the system, from after failure injection to after rollback, and the prior state of the system in the second test case relating to the failure test of the system. This makes it possible to identify the execution order of the test cases based on the similarity information between the intermediate state of the system in the first consecutive test case from after failure injection to after rollback and the prior state of the system in the subsequent second test case.

[0080] Furthermore, as explained in each item of Figure 9 and in Figure 16, the above-mentioned base test execution support system has a test case management table (Figure 9) that manages the execution time of the test, including the time required for the rollback (for example, abnormal injection 922 and rollback 923 in execution time 908 in Figure 9), and a test constraint management table (Figure 10) that manages the upper limit of the total execution time of the test. The processor identifies a subset of test cases that does not exceed the upper limit of the total execution time from the execution order of the identified test cases (Figures 16 and 11). This makes it possible to identify the execution order of test cases within a range that does not exceed the upper limit of the total execution time.

[0081] Furthermore, as explained in each item of Figure 9 and in Figure 17, the above-mentioned base test execution support system has a test case management table (Figure 9) that manages the execution fees for the above tests (for example, the execution fee 907 in Figure 9). The processor calculates the execution fee for the identified test case based on predetermined fees, reflects the calculated execution fee in the test case management table, and also reflects the execution fee in the priority of the test case. This makes it possible to determine the execution order of the test cases according to the execution fee of each test case.

[0082] Furthermore, as explained in each item of Figure 9 and in Figure 16, the above-mentioned base test execution support system has a test case management table (Figure 9) that manages the execution fee for the above-mentioned tests (for example, the execution fee 907 in Figure 9), and a test constraint management table (Figure 10) that manages the upper limit of the total execution fee for the above-mentioned tests. The processor identifies a subset of test cases that does not exceed the upper limit of the total execution fee from the execution order of the identified test cases (Figures 16 and 11). This makes it possible to identify the execution order of test cases within a range that does not exceed the upper limit of the total execution fee.

[0083] Furthermore, as explained in each item of Figure 9 and in Figure 16, the above-mentioned base test execution support system has the above-mentioned test case management table (Figure 9) which manages the success or failure history of past test executions (for example, success 920, failure 921 in Figure 9), and the above-mentioned processor reflects the execution results of past tests in the success or failure history and also reflects the success or failure history in the priority of the test cases. This makes it possible to determine the execution order of test cases based on the success or failure history of the tests.

[0084] Furthermore, as explained in S1303-1306 in Figure 13, the processor generates test patterns for injected faults, pre-fault states, and post-fault states based on the fault injection targets included in the temporary system configuration information management table (Figure 7) showing the system configuration, the fault definition table (Figure 8A) defining resource failures in the system, and the failure behavior definition table (Figure 8B) defining the behavior of the resource to which the fault has been injected. Based on the pre-fault state and post-fault state in the generated test patterns, the processor identifies a method for initializing the system or a method for rolling back the database of the system, and defines it as a test case. If, in testing past test cases, the pre-fault state and post-fault state are similar to a certain extent, the execution fee and execution time of the test of the similar past test case are used as the execution fee and execution time of the test of the identified test case, and the test case management table (for example, execution fee 907, execution time 908 in Figure 9) is updated. This makes it possible to determine the execution fee and execution time based on the performance of past test cases.

[0085] Furthermore, as explained in Figures 11, 15, and 17, the above-mentioned base test execution support system has a pair temporary management table (Figure 16) that stores the execution order of the two or more identified test cases. The processor stores in the pair temporary management table any skippable processes between two consecutive test cases based on the similarity information for the identified test cases in the execution order. When executing the test cases, the processor skips the skippable processes based on the information indicating the skippable processes and executes the test. This makes it possible to execute a test that shortcuts the skippable processes between two consecutive test cases.

[0086] The present invention is not limited to the embodiments described above, and in implementation, the components can be modified and implemented without departing from the gist of the invention, or multiple components disclosed in the embodiments can be appropriately combined. For example, multiple management servers 105 can be set up, and the tests of the test cases described in the embodiment can be executed in parallel in these multiple environments. After determining the execution order of all test cases in each environment, the execution order can be determined for pairs (subsets) of two test cases stored in the pair temporary management table 1600 shown in Figure 16 so that the estimated execution time of the tests is roughly equal. For example, the estimated execution time of the tests can be averaged across each management server 105 by swapping the subset stored in one management server 105 with the subset stored in another management server 105. [Explanation of Symbols]

[0087] 1 System 101 IT Systems 102 IT System Management Server 103 Source Code Repository 105 Management Server 106 Console 104 Network 601 System Configuration Information Temporary Management Table 602 Test Definition Various Tables 603 Test Case Generation Unit 604 Test Case Management Table 605 Test Case Execution Order Identification Unit 606 Test Constraint Management Table 607 Test Case Execution Order Management Table 608 Test Case Execution Unit

Claims

1. A foundational test execution support system that assists in performing tests related to the foundation of a system, which is executed by a computer having a processor and memory, The aforementioned processor, Similarity information of system states between two test cases, wherein the execution order of two or more test cases is identified based on the similarity information between the intermediate state of the system in the first test case and the prior state of the system in the second test case. A foundational test execution support system characterized by the following features.

2. The aforementioned processor, The identification is performed based on similarity information of the system state, which includes one or more of the following elements: (1) a state indicating whether the database possessed by the system is the primary or secondary component; (2) a state indicating whether writing to the system is permitted or in progress; (3) a state indicating whether the database is running or not; and (4) the number of databases. The base test execution support system according to feature 1.

3. The aforementioned processor, An index indicating the similarity of the system state between the two test cases is defined by one of the following: the execution time reduced when the two test cases are executed consecutively, a value that takes into account whether some processing can be skipped when they are executed consecutively, or the execution time when skippable processing is skipped during consecutive execution. The base test execution support system according to feature 1.

4. The aforementioned base test execution support system is The aforementioned system has a test case management table for managing the time required for database rollback, The aforementioned processor, Based on the similarity information between the intermediate state of the system in the first test case relating to the failure test of the system, from after fault injection to after rollback, and the prior state of the system in the second test case relating to the failure test of the system, the identification is performed. The base test execution support system according to feature 1.

5. The aforementioned base test execution support system is The system includes a test case management table that manages the execution time of the tests, including the time required for the rollback, and a test constraint management table that manages the upper limit of the total execution time of the tests. The aforementioned processor, From the execution order of the identified test cases, a subset of test cases is identified that does not exceed the upper limit of total execution time. The base test execution support system according to feature 4.

6. The aforementioned base test execution support system is The system has a test case management table that manages the execution fees for the aforementioned tests, The aforementioned processor, Based on predetermined fees, the execution fee for the identified test case is calculated, the calculated execution fee is reflected in the test case management table, and the execution fee is reflected in the priority of the test case. The base test execution support system according to feature 4.

7. The aforementioned base test execution support system is The system includes a test case management table that manages the execution fees for the tests, and a test constraint management table that manages the upper limit of the total execution fees for the tests. The aforementioned processor, From the execution order of the identified test cases, identify a subset of test cases that does not exceed the upper limit of the total execution cost. The base test execution support system according to feature 4.

8. The aforementioned base test execution support system is The test case management table manages the success or failure history of past test executions, The aforementioned processor, The results of past tests are reflected in the success history or failure history, and the success history or failure history is reflected in the priority of the test cases. The base test execution support system according to feature 4.

9. The aforementioned processor, Based on the system configuration information temporary management table showing the system configuration, which contains the fault injection targets, the fault definition table defining resource failures in the system, and the fault behavior definition table defining the behavior of the resource to which the fault has been injected, test patterns are generated for the injected fault, the pre-fault state, and the post-fault state. Based on the pre-failure state and post-failure state in the generated test pattern, a method for initializing the system or a method for rolling back the database of the system is identified and designated as the test case. In testing past test cases, if the pre-failure state and the post-failure state are similar to a certain extent, the test case management table is updated with the execution fee and execution time of the similar past test case as the execution fee and execution time of the identified test case. The base test execution support system according to feature 4.

10. The aforementioned base test execution support system is The system has a pair temporary management table that stores the execution order of the two or more specified test cases, The aforementioned processor, For the test cases of the specified execution order, based on the similarity information, skippable processes between two consecutive test cases are stored in the pair temporary management table. When executing the aforementioned test case, the skippable processes are skipped and the test is executed based on the information indicating the skippable processes. The base test execution support system according to feature 1.

11. A method for supporting the execution of foundational tests, which is performed by a computer having a processor and memory, and which assists in the execution of foundational tests of a system. The aforementioned computer, Similarity information of system states between two test cases, wherein the execution order of two or more test cases is identified based on the similarity information between the intermediate state of the system in the first test case and the prior state of the system in the second test case. A method for supporting the execution of foundational tests, characterized by the following features.

Images