Test support device and test support method

Abstract

To lessen a burden of selecting a test case for conducting a test on a user system even in a case where the user system is formed of a plurality of resources.SOLUTION: A storage unit 102 stores a plurality of registration configuration data sets concerning the configurations of a plurality of user systems in association with respective test cases in which tests on the user systems corresponding to the registration configuration data sets were conducted. A similarity calculation unit 106 calculates a data similarity which indicates the similarity between the input configuration data set concerning the configuration of a target system which is a user system to be tested, and each of the registration configuration data sets. Extraction units (a score calculation unit 107 and a test case extraction unit 108) extract a candidate test case which is a candidate of a test case for conducting a test on the target system, from the storage unit on the basis of the corresponding data similarity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a test support device and a test support method. [Background technology]

[0002] In recent years, virtualization of IT (Information Technology) resources has been progressing in order to improve computer efficiency, such as improving server maintenance performance and portability of execution environments. Technologies for virtualizing IT resources include virtual servers, which logically divide a physical server to function as multiple servers, and containers, which logically divide an OS (Operating System) running on a server into process units to function as a single server. Orchestration tools have also been developed to efficiently manage and operate virtualized IT resources. In conjunction with this, IaC (Infrastructure as Code), which manages IT resources and services using source code and enables the construction and operation of user systems on which applications run, has been attracting attention as a technology for effectively using orchestration tools.

[0003] An orchestration tool manages a user system that is composed of IT resources, using system configuration data that describes the system configuration in code. The IT resources managed by the system configuration data are diverse and include source codes of application programs, container images, application programs, libraries, and binaries, and in recent years, they have tended to become larger in scale.

[0004] Furthermore, in order to ensure that the user system operates without any problems, unit tests of IT resources are usually performed in a development environment before the user system is built in a production environment. In unit tests, for example, source code validation checks, source code function checks, and container image vulnerability checks are performed. On the other hand, verification of usability and fault tolerance cannot be performed sufficiently in the development environment, so these are not performed in unit tests, but are performed at the system test stage in a staging environment or the like. Note that unit tests are often automated with a CI (Continuous Integration) tool to reduce the burden on users. System configuration data that has passed unit tests using a CI tool is deployed to a staging environment by a CD (Continuous Delivery) tool, and system tests are performed there.

[0005] Regarding usability and fault tolerance, which are verified in system testing, it is becoming difficult to verify them with conventional comprehensive testing because user systems are composed of a combination of multiple resources. For this reason, exploratory testing is often used. However, in exploratory testing, test cases must be selected based on the current state of the user system and the user's experience.

[0006] In addition, when constructing a user system using an orchestration tool, it is necessary to change system configuration data, etc., in accordance with an upgrade of the orchestration tool and functional modifications of the application. For example, when upgrading the orchestration tool, it is necessary to change the system configuration data according to instructions from the community or cloud vendor. When the system configuration data is changed, the user system may be affected, for example, programs that depend on the changes may not operate properly, or compatibility with the OS and middleware may deteriorate, resulting in reduced performance, and therefore unit tests and system tests must be performed again. In addition, since the user system managed by the orchestration tool is composed of multiple resources, it is also necessary to test whether or not a problem caused by the update of one resource has occurred in other resources and the entire system. However, if tests using all test cases are performed every time a small number of resources are changed, the user's man-hours will increase, which is inefficient.

[0007] For this reason, it has been considered to suggest test cases to be executed to the user. For example, when a part of source code is changed, there is no need to execute tests on parts that are not dependent on the change, so by suggesting test cases to the user according to the change in the source code, it becomes possible to execute tests efficiently. However, in order to accurately grasp other parts that are affected by a change in a part of source code, it is necessary to accurately grasp the dependency relationships within the source code, which imposes a large burden on selecting test cases to be executed.

[0008] In response to this, Patent Document 1 discloses a software testing system that identifies changes in software source code and the scope of their impact, and generates test cases limited to that scope. This makes it possible to reduce the burden of selecting test cases for tests accompanying source code changes. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2015-69400 A Summary of the Invention [Problem to be solved by the invention]

[0010] However, since the user system managed by the orchestration tool is composed of multiple resources, it is difficult to identify the impact range across multiple resources from the dependency of the source code. Also, although the consistency of the source code is guaranteed, malfunction or performance degradation may occur depending on the compatibility between resources. In such cases, it is difficult to identify the impact range from the changes in the software source code, so the technology described in Patent Document 1 makes it difficult to reduce the burden associated with selecting test cases.

[0011] An object of the present invention is to provide a test support device and a test support method that can reduce the burden associated with selecting test cases for testing a user system, even when the user system is composed of multiple resources. [Means for solving the problem]

[0012] A test support device according to one aspect of the present disclosure has a memory unit that stores each of multiple registered configuration data related to the configuration of multiple user systems in association with a test case when a test of the user system corresponding to the registered configuration data is conducted, a similarity calculation unit that calculates data similarity, which is the similarity between input configuration data related to the configuration of a target system, which is the user system to be tested, and each of the multiple registered configuration data, an extraction unit that extracts candidate test cases that are candidates for test cases when a test of the target system is conducted from the memory unit based on each data similarity, and an output unit that outputs the candidate test cases. Effect of the Invention

[0013] According to the present invention, even when a user system is configured with a plurality of resources, it is possible to reduce the burden associated with selecting test cases for testing the user system. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a diagram illustrating a functional configuration of an exploratory testing support device according to a first embodiment of the present disclosure. [Diagram 2] FIG. 11 is a diagram illustrating an example of registered configuration data. [Diagram 3] FIG. 11 is a diagram illustrating an example of linking information. [Figure 4] 11 is a flowchart illustrating an example of a similarity calculation process. [Diagram 5] 11A and 11B are diagrams for explaining an example of input configuration data and registered configuration data; [Figure 6] 11 is a flowchart illustrating an example of a score calculation process. [Figure 7] 11 is a flowchart illustrating an example of a test case extraction process. [Figure 8] FIG. 11 is a diagram illustrating a functional configuration of an exploratory testing support device according to a second embodiment of the present disclosure. [Figure 9] 13 is a flowchart illustrating another example of the test case extraction process. [Figure 10] 13 is a flowchart illustrating an example of a common element extraction process. [Figure 11] FIG. 13 is a diagram illustrating an example of a test case candidate display screen. [Figure 12] FIG. 11 is a diagram illustrating a functional configuration of an exploratory testing support device according to a third embodiment of the present disclosure. [Figure 13] 13 is a flowchart illustrating another example of the test case extraction process. [Figure 14] FIG. 11 is a diagram for explaining an example of a filtering process. [Figure 15] FIG. 13 is a diagram illustrating an example of an input screen. [Figure 16] FIG. 2 illustrates an example of a hardware configuration of an exploratory testing support device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following embodiments will be described with reference to the drawings. The following are merely examples of the embodiments, and the invention itself is not intended to be limited to the specific contents described below. In the following description, the same or similar elements and processes may be given the same reference numerals, and duplicated descriptions may be omitted. In the following embodiments, differences from the previous embodiments may be mainly described, and duplicated descriptions may be omitted. The following description of the embodiments and the configurations and processes shown in the drawings are intended to provide an overview of the embodiments to the extent necessary for understanding and implementing the present invention, and are not intended to limit the implementation of the present invention. In addition, each embodiment and each modified example may be combined in part or in whole within a range that is consistent with each other without departing from the spirit of the present invention. EXAMPLES

[0016] Fig. 1 is a diagram showing a functional configuration of an exploratory test support device according to a first embodiment of the present disclosure. The exploratory test support device 100 shown in Fig. 1 is a test support device for selecting candidate test cases, which are candidates for test cases when testing a user system configured with multiple resources managed by an orchestration tool, so that a user can select a test case to be actually performed in the test from the candidate test cases. In this embodiment, the test of the user system is an exploratory test.

[0017] 1, the exploratory test support device 100 is communicatively connected to a user terminal 111 and a CI tool 112. The user terminal 111 is communicatively connected to a CD tool 114 and an exploratory test tool 116, and the CI tool 112 is communicatively connected to a repository 113. The repository 113 is communicatively connected to the CD tool 114. The CD tool 114 and the exploratory test tool 116 are communicatively connected to an application execution environment 115.

[0018] The user terminal 111 is a terminal device used by a user who uses the exploratory testing support device 100. For example, in response to an operation from the user, the user terminal 111 inputs information used for selection of candidate test cases by the exploratory testing support device 100 to the exploratory testing support device 100. Also, for example, in response to an operation from the user, the user terminal 111 selects an implementation test case, which is a test case to be used in an actual test, from the candidate test cases selected by the exploratory testing support device 100.

[0019] The CI tool 112 performs continuous integration. For example, the CI tool 112 acquires system configuration data related to the configuration of a user system to be tested from the repository 113 and inputs the data to the exploratory test support device 100. The repository 113 stores a plurality of system configuration data for constructing the user system. Note that the system configuration data may be composed of a plurality of files.

[0020] The CD tool 114 performs continuous delivery. For example, the CD tool 114 obtains system configuration data from the repository 113, and instantiates and builds a user system according to the system configuration data. The application execution environment 115 is hardware for building a user system, and for example, executes an application 1151 on a user system built according to the system configuration data.

[0021] The exploratory test tool 116 executes an exploratory test in the application execution environment 115 according to the implementation test case selected on the user terminal 111 .

[0022] The CI tool 112, the repository 113, the CD tool 114, the application execution environment 115, and the exploratory test tool 116 can be mounted on a computer server or the like, and in this case, may be mounted on the same computer server or the like.

[0023] The exploratory testing support device 100 includes a system configuration data input unit 101, a memory unit 102, a user input unit 103, a weighting setting unit 104, a threshold setting unit 105, a similarity calculation unit 106, a score calculation unit 107, a test case extraction unit 108, a test case output unit 109, and a stored data update unit 110.

[0024] The system configuration data input unit 101 is an input unit that accepts system configuration data relating to the configuration of a target system, which is a user system to be tested, as input configuration data. In addition, the system configuration data input unit 101 converts the input configuration data into a specified data format. In this embodiment, as described below, the input configuration data is compared with registered configuration data, which is system configuration data stored in the storage unit 102, so the system configuration data input unit 101 converts the input configuration data into the same data format as the registered configuration data so that this comparison is possible.

[0025] The system configuration data may be input to the system configuration data input unit 101 from a user terminal 111 in response to a user operation, or may be obtained from a repository 113 by a CI tool 112 and input to the system configuration data input unit 101.

[0026] The storage unit 102 stores various information. Specifically, the storage unit 102 includes a system configuration data storage unit 1021, a test case storage unit 1022, and an association information storage unit 1023.

[0027] The system configuration data storage unit 1021 stores registered configuration data, which is system configuration data to be compared with the system configuration data input to the system configuration data input unit 101. The registered configuration data is, for example, system configuration data relating to a user system on which a test has been conducted in the past, and is registered in advance by a user. In addition, when the exploratory testing support device 100 is in operation, the registered configuration data may be added or updated by the stored data update unit 110.

[0028] Fig. 2 is a diagram showing an example of registered configuration data. In Fig. 2, a plurality of system configuration definition file groups 200A to 200C are shown as the registered configuration data. In the following, when there is no need to distinguish between the system configuration definition file groups 200A to 200C, the system configuration definition file groups 200A to 200C may be collectively referred to as the system configuration definition file group 200.

[0029] The system configuration definition file group 200 is a file group for defining resources that constitute a user system. Examples of resources include applications, networks, and databases. Each system configuration definition file 201 included in the system configuration definition file group 200 has one or more items that define the configuration of the user system. The items indicate information required to operate the resources on the application execution environment 115. For example, the items include the name, version information, function information, and storage method of the resource, the port number to be used, and the like. For example, if the item is version information of an API (Application Programming Interface), the version information of the actual API used in the user system is stored as the item value.

[0030] By entering the necessary contents in each item of the system configuration data, the user of the orchestration tool can instantiate and execute the user system in the application execution environment 115. Therefore, the system configuration data of user systems having similar configurations are characterized by being similar to each other.

[0031] Returning to the explanation of Fig. 1, the test case storage unit 1022 stores a plurality of test cases 10221 that define the contents of tests to be performed on the user system. The test cases 10221 define, for example, a script for performing a test, prerequisites and operation procedures for performing a test, a method for checking test results, etc. The test cases for tests to be actually performed are determined according to, for example, items of system configuration data, etc.

[0032] The linking information storage unit 1023 stores linking information linking each piece of registered configuration data with a test case 10221 used when an exploratory test of the user system corresponding to the registered configuration data was performed.

[0033] 3 is a diagram showing an example of the linking information 300. The linking information 300 shown in FIG.

[0034] The test case field 301 stores a test case name that identifies the test case 10221. The system configuration data field 302 stores a configuration data name that identifies registered configuration data linked to the test case 10221 of the test case name in the test case field 301. The weighting value field 303 stores a weighting value corresponding to each item included in the registered configuration data of the configuration data name in the system configuration data field 302. The threshold field 304 stores a threshold corresponding to the registered configuration data of the configuration data name in the system configuration data field 302.

[0035] As described below, the weighting values ​​and threshold values ​​are used to select candidate test cases from the test cases 10221 stored in the test case memory unit 1022, which are candidates for test cases to conduct exploratory testing of the user system using the input configuration data accepted by the system configuration data input unit 101.

[0036] The user input unit 103 receives weighting values ​​and thresholds to be set in the association information storage unit 1023 from the user terminal 111. The user input unit 103 also receives a selection result from the user terminal 111 of an implemented test case selected from the candidate test cases.

[0037] The weight setting unit 104 sets the weight value received by the user input unit 103 in the linking information in the linking information storage unit 1023. The threshold setting unit 105 sets the threshold value received by the user input unit 103 in the linking information in the linking information storage unit 1023.

[0038] The similarity calculation unit 106 compares the input configuration data received by the system configuration data input unit 101 with each registered configuration data stored in the system configuration data storage unit 1021, and calculates data similarity, which is the similarity between the input configuration data and each registered configuration data. Specifically, the similarity calculation unit 106 compares, for each item of the input configuration data, the item value of the input configuration data with the item value of each registered configuration data, and calculates item similarity, which is the similarity of the item value of each item, as the data similarity between the input configuration data and each registered configuration data.

[0039] In this embodiment, the item values ​​are character strings, and the item similarity is a character string distance that represents the similarity between the character strings that are the item values. The character string distance can be calculated using, for example, Gestalt pattern matching, the Levenshtein distance method, or the Jaro-Winkler distance method. The similarity may also be normalized so that it becomes a predetermined value (for example, 100%) when each item value is completely the same.

[0040] The score calculation unit 107 and the test case extraction unit 108 constitute an extraction unit that extracts candidate test cases from a plurality of test cases 10221 stored in the test case memory unit 1022 based on the configuration similarity (item similarity of each item) calculated by the similarity calculation unit 106.

[0041] Specifically, the score calculation unit 107 calculates the score of each test case 10221 based on the weighting value of each item associated with each test case 10221 in the association information stored in the association information storage unit 1023 and the item similarity for each item calculated by the similarity calculation unit 106. Specifically, the score calculation unit 107 calculates the weighted sum of each item similarity (i.e., the sum of values ​​obtained by multiplying the item similarity by the weighting value for each item) as the score.

[0042] The test case extraction unit 108 compares the score of each test case 10221 calculated by the score calculation unit 107 with the threshold value of each test case 10221 in the linking information stored in the linking information storage unit 1023, and selects a candidate test case from the test cases 10221. Specifically, the test case extraction unit 108 extracts test cases 10221 whose scores are equal to or greater than the threshold value, and selects them as candidate test cases.

[0043] Therefore, in this embodiment, the larger the weighting value, the easier it is for the test case 10221 to be selected as a candidate test case. The score of the test case 10221 is a numerical value that expresses the ease of selecting the test case 10221 as a candidate test case, and the larger the score, the easier it is for the test case 10221 to be selected as a candidate test case. The threshold is a numerical value that is a criterion for selecting a candidate test case. The higher the threshold, the more likely it is that only the test case 10221 with a high score will be selected as a candidate test case, that is, if the item of the system configuration data linked to the test case 10221 is strongly similar to the item of the input configuration data, the test case 10221 will be selected as a candidate test case. Conversely, the lower the threshold, the more likely it is that a test case with a relatively low score will be selected as a candidate test case, that is, even if the item of the system configuration data linked to the test case 10221 is relatively weakly similar to the item of the input configuration data, the test case 10221 will be selected as a candidate test case.

[0044] The test case output unit 109 outputs the candidate test cases selected by the test case extraction unit 108 to the user terminal 111. The user terminal 111 displays the candidate test cases, and when one of the candidate test cases is selected as an implementation test case, the user terminal 111 notifies the implementation test case to the exploratory test tool 116 to perform an exploratory test. The user terminal 111 also inputs the selection result to the system configuration data input unit 101 of the exploratory test support device 100.

[0045] The stored data update unit 110 is an update unit that updates at least one of the weighting value and the threshold value included in the association information in the association information storage unit 1023 based on the selection result received by the system configuration data input unit 101 .

[0046] Fig. 4 is a flowchart for explaining an example of a similarity calculation process for calculating the similarity between the input configured data and the registered configured data by the similarity calculation unit 106. Fig. 5 is a diagram for explaining an example of the input configured data and the registered configured data.

[0047] In the similarity calculation process, first, the user input unit 103 accepts the system configuration data input from the CI tool 112 as the input configuration data (step S401). For example, the user input unit 103 accepts the system configuration data 501 (system configuration definition file group X) as the input configuration data. The system configuration data 501 includes a system configuration definition file 502 (system configuration definition file X-1) and a system configuration definition file 503 (system configuration definition file group X-2).

[0048] Next, the similarity calculation unit 106 acquires, as the evaluated configuration definition file, one of the system configuration definition files included in the input configuration data accepted by the user input unit 103 (step S402). Here, the similarity calculation unit 106 acquires, as the evaluated configuration definition file, the system configuration definition file that was first input as the input configuration data. For example, the similarity calculation unit 106 acquires the system configuration definition file 502 shown in FIG. 5 as the evaluated configuration definition file.

[0049] The similarity calculation unit 106 acquires one of the registered configuration data from the system configuration data storage unit 1021 as configuration data to be compared (step S403). Here, the similarity calculation unit 106 acquires the oldest registered configuration data as the configuration data to be compared. For example, the similarity calculation unit 106 acquires the registered configuration data 511 (system configuration definition file group A) from the registered configuration data 511 to 541 shown in FIG. 5 as the configuration data to be compared. In this embodiment, "oldest" refers to the oldest date and time of a predetermined type of timestamp (e.g., update date and time) assigned to the file.

[0050] The similarity calculation unit 106 acquires one of the system configuration definition files included in the comparison target configuration data as a comparison target definition file (step S404). Here, the similarity calculation unit 106 acquires the oldest system configuration definition file as a comparison target definition file. For example, the similarity calculation unit 106 acquires the system configuration definition file 512 shown in FIG. 5 as a comparison target definition file.

[0051] The similarity calculation unit 106 acquires one of the items included in the evaluation configuration definition file as the evaluation item (step S405). Here, the similarity calculation unit 106 acquires the first item described in the evaluation configuration definition file as the evaluation item. For example, the similarity calculation unit 106 acquires item 1 shown in FIG. 5 as the evaluation item.

[0052] The similarity calculation unit 106 compares the item value of the evaluation item with the item value of the same item that is the same item as the evaluation item among the items included in the comparison target definition file, and calculates the similarity between the evaluation item and the same item (step S406). For example, in the case of Fig. 5, the similarity calculation unit 106 compares the item value X1-1 of item 1 of the system configuration definition file 502, which is the evaluation item, with the item value A1-1 of item 1 of the system configuration definition file 512, which is the comparison target definition file. Note that if the same item that is the same item as the evaluation item does not exist in the comparison target definition file, the similarity calculation unit 106 skips the process of step S406 and excludes the evaluation item from the calculation of the similarity.

[0053] The similarity calculation unit 106 judges whether or not there is an item that has not been acquired as an evaluation item in the evaluation configuration definition file (step S407).

[0054] If there are any items that have not been acquired (step S407: Yes), the similarity calculation unit 106 acquires the items that have not been acquired as evaluation items among the items included in the evaluation configuration definition file (step S408), and returns to the processing of step S406. Here, the similarity calculation unit 106 acquires the first described item and item value among the items that have not been acquired.

[0055] If there is no item that has not been acquired (step S407: No), the similarity calculation unit 106 determines whether or not there is a system configuration definition file that has not been acquired as a comparison target definition file in the comparison target configuration data (step S409).

[0056] If there is a comparison target definition file that has not been acquired (step S409: Yes), the similarity calculation unit 106 acquires one of the system configuration definition files included in the comparison target configuration data that has not been acquired as a comparison target definition file (step S410), and returns to the process of step S405. Here, the similarity calculation unit 106 acquires the oldest system configuration definition file among the system configuration definition files that have not been acquired as a comparison target definition file.

[0057] If there is no comparison target definition file that has not been acquired (step S409: No), the similarity calculation unit 106 determines whether there is registered configuration data that has not been acquired as comparison target configuration data in the system configuration data storage unit 1021 (step S411).

[0058] If there is registered configuration data that has not been acquired (step S411: Yes), the similarity calculation unit 106 acquires one of the registered configuration data that has not been acquired yet among the registered configuration data stored in the system configuration data storage unit 1021 as configuration data to be compared (step S412), and returns to the processing of step S404. Here, the similarity calculation unit 106 acquires the oldest registered configuration data among the registered configuration data that has not been acquired yet as configuration data to be compared.

[0059] If there is no registered configuration data that has not been acquired (step S411: No), the similarity calculation unit 106 determines whether or not there is a system configuration definition file in the input configuration data that has not been acquired as an evaluated configuration definition file (step S412).

[0060] If there is a system configuration definition file that has not been acquired (step S412: Yes), the similarity calculation unit 106 acquires one of the system configuration definition files included in the input configuration data that has not yet been acquired as an evaluated configuration definition file (step S413), and returns to the processing of step S403.

[0061] If there is no system configuration definition file that has not been acquired (step S412: No), the similarity calculation unit 106 ends the similarity calculation process.

[0062] 6 is a flowchart for explaining an example of a score calculation process in which the score calculation unit 107 calculates the score of the test case 10221. The score calculation process is executed upon completion of the similarity calculation process described with reference to FIGS.

[0063] In the score calculation process, first, the score calculation unit 107 acquires one of the test cases 10221 from the test case storage unit 1022 (step S601). The score calculation unit 107 refers to the linking information in the linking information storage unit 1023, and acquires, as a target item, one of the items included in the system configuration definition file in the registered configuration data (system configuration definition file group) associated with the acquired test case 10221 (step S602).

[0064] The score calculation unit 107 obtains a weighting value corresponding to the target item from the linking information in the linking information storage unit 1023 (step S603). For example, when the item 1 shown in FIG. 3 is obtained as the target item in step S602, the score calculation unit 107 obtains a weighting value W A1 Get the.

[0065] The score calculation unit 107 multiplies the item similarity corresponding to the target item calculated by the similarity calculation unit 106 (item similarity between the target item and an item that is the same as the target item among the items of the input configuration data) by the acquired weighting value (step S604).

[0066] The score calculation unit 107 determines whether or not there is an item that has not been acquired as a target item in the system configuration definition file in the registered configuration data linked to the acquired test case 10221 (step S605).

[0067] If there are items that have not been acquired (step S605: Yes), the score calculation unit 107 acquires one of the items that has not yet been acquired among the items included in the system configuration definition file in the registered configuration data linked to the acquired test case 10221 as the target item (step S606), and returns to the processing of step S603.

[0068] If there is no item that has not been acquired (step S605: No), the score calculation unit 107 calculates and holds the sum of the item similarities multiplied by the weighting value as the score of the acquired test case 10221 (step S607). For example, the score of the test case A shown in FIG. 3 is calculated by multiplying the item similarity of item 1 by D A1 , the item similarity of item 2 is D A2 Then, D A1 ×W A1 +D A2 ×W A2 It becomes.

[0069] The score calculation unit 107 judges whether or not there is a test case 10221 that has not been acquired in the test case storage unit 1022 (step S608).

[0070] If there is a test case 10221 that has not been acquired (step S608: Yes), the score calculation unit 107 acquires one of the test cases 10221 that has not yet been acquired from among the test cases 10221 stored in the test case memory unit 1022 (step S609), and returns to the processing of step S602.

[0071] If there is no unacquired test case 10221 (step S608: No), the score calculation section 107 ends the score calculation process.

[0072] 7 is a flowchart for explaining an example of a test case extraction process for selecting candidate test cases by the test case extraction unit 108 etc. The test case extraction process is executed upon completion of the score calculation process described with reference to FIG.

[0073] In the test case extraction process, first, the test case extraction unit 108 acquires one of the test cases 10221 from the test case storage unit 1022 (step S701). Here, the test case extraction unit 108 acquires the oldest test case 10221 from the test case storage unit 1022. For example, the test case extraction unit 108 acquires the test case 10221 corresponding to the test case A in FIG.

[0074] The test case extraction unit 108 acquires the score of the acquired test case 10221 from among the scores calculated by the score calculation unit 107 in S607 (step S702). Hereinafter, the acquired score will be referred to as S A It is sometimes written as follows.

[0075] The test case extraction unit 108 acquires a threshold value corresponding to the acquired test case 10221 from the association information in the association information storage unit 1023 (step S703). For example, the test case extraction unit 108 acquires a threshold value T A Get the.

[0076] The test case extraction unit 108 extracts the score of the acquired test case 10221 from the test case 10221 if the score is equal to or greater than a threshold (for example, S A ≧T A ) (step S704).

[0077] If the score is equal to or greater than the threshold (step S704: Yes), the test case extraction unit 108 validates the acquired test case 10221 (step S705). On the other hand, if the score is less than the threshold (step S704: No), the test case extraction unit 108 invalidates the acquired test case 10221 (step S706). Note that the validation and invalidation of the test cases 10221 are performed, for example, by setting a flag for the validated test case 10221 and not setting a flag for the invalidated test case 10221.

[0078] After that, the test case extraction unit 108 judges whether or not there is a test case 10221 that has not been acquired in the test case storage unit 1022 (step S707).

[0079] If there is a test case 10221 that has not been acquired (step S707: Yes), the test case extraction unit 108 acquires one of the test cases 10221 that has not yet been acquired from among the test cases 10221 stored in the test case memory unit 1022 (step S708), and returns to the processing of step S702.

[0080] If there is no unacquired test case 10221 (step S707: No), the test case extraction unit 108 outputs the enabled test case 10221 as a candidate test case to the user terminal 111 via the test case output unit 109 (step S709). At this time, the user terminal 111 displays, for example, a list of candidate test cases. The user selects an implementation test case for actually performing an exploratory test from the list of candidate test cases. Then, the user terminal 111 outputs the user's selection result to the exploratory test support device 100.

[0081] Thereafter, when the user input unit 103 receives the user's selection result from the user terminal 111, the stored data update unit 110 updates the weighting value of the linking information in the linking information storage unit 1023 based on the selection result (step S710), and terminates the test case extraction process.

[0082] In updating the weighting values, the stored data update unit 110, for example, determines that the weighting values ​​of each item associated with the candidate test case not selected as the executed test case were not appropriate, and lowers the weighting values. At this time, the stored data update unit 110 may not update the weighting values ​​of each item associated with the candidate test case selected as the executed test case, or may further increase the weighting values. In addition, the stored data update unit 110 may display a screen for updating the weighting values ​​of each item associated with each candidate test case on the user terminal 111. In this case, the stored data update unit 110 updates the weighting values ​​according to the contents input by the user to the user terminal 111 according to the screen. Note that the stored data update unit 110 may update the thresholds instead of or in addition to the weighting values. In updating the thresholds, the stored data update unit 110, for example, determines that the thresholds of each item associated with the candidate test case not selected as the executed test case were not appropriate, and raises the thresholds.

[0083] As described above, according to this embodiment, the storage unit 102 stores each of a plurality of registered configuration data related to the configuration of a plurality of user systems in association with the test case 10221 when a test of the user system corresponding to the registered configuration data is performed. The similarity calculation unit 106 calculates data similarity, which is the similarity between the input configuration data related to the configuration of the target system, which is the user system to be tested, and each of the plurality of registered configuration data. The extraction unit (score calculation unit 107 and test case extraction unit 108) extracts candidate test cases, which are candidates for the test case 10221 when a test of the target system is performed, from the storage unit based on each data similarity. Therefore, since the test case 10221 effective for the system configuration data of the configuration of the user system to be tested can be extracted as the candidate test case, it is possible to reduce the burden associated with selecting a test case for testing the user system even when the user system is configured with a plurality of resources.

[0084] In this embodiment, the similarity calculation unit 106 calculates, for each item of the input configuration data, an item similarity between the item value of the item and the item value of the same item in the registered configuration data as the data similarity. In this case, it is possible to more appropriately calculate the similarity between the input configuration data and the registered configuration data, thereby improving the accuracy of selecting candidate test cases.

[0085] In this embodiment, the score calculation unit 107 calculates a score, which is a weighted sum of the similarity of each item between the registered configuration data and the input configuration data, for each test case associated with the registered configuration data. The test case extraction unit 108 extracts candidate test cases based on the score. This makes it possible to more appropriately select candidate test cases.

[0086] In addition, in this embodiment, the test case extraction unit 108 extracts test cases whose scores are equal to or greater than a threshold as candidate test cases, which makes it possible to more appropriately select candidate test cases.

[0087] Furthermore, in this embodiment, since the threshold is set for each test case, it becomes possible to more appropriately select candidate test cases.

[0088] In this embodiment, the user input unit 103 accepts a selection result of selecting one of the candidate test cases. The stored data update unit 110 updates at least one of the weighting value and the threshold value based on the selection result. Therefore, it is possible to continuously change at least one of the weighting value and the threshold value based on the user's selection, which makes it possible to further increase the accuracy of the selection of the candidate test cases. EXAMPLES

[0089] In Example 1, an example was described in which candidate test cases were selected and presented to the user. In this example, an example is described in which, in addition to the candidate test cases, the reasons for selecting the candidate test cases are also presented.

[0090] Fig. 8 is a diagram showing a functional configuration of an exploratory testing support device according to a second embodiment of the present disclosure. The exploratory testing support device 100 shown in Fig. 8 is different from the exploratory testing support device 100 according to the first embodiment shown in Fig. 1 in that it further includes a common element extraction unit 800.

[0091] The common element extraction unit 800 extracts, from the linking information in the linking information storage unit 1023, a common element of the registered configuration data corresponding to the test case 10221 enabled by the test case extraction unit 108 as a selection reason. Specifically, a common element is an item whose item similarity with the item value of an item of the input configuration data is equal to or greater than a predetermined value. In this embodiment, the predetermined value is "100%". In other words, a common element is an item whose item value is the same as that of the input configuration data.

[0092] 9 is a flowchart for explaining a test case extraction process including an element extraction process for extracting a common element. In the following, the parts different from the test case extraction process of the first embodiment explained with reference to FIG. 7 will be mainly explained.

[0093] First, the processes of steps S701 to S708 are performed. If it is determined in step S707 that there is no unacquired test case 10221 (step S707: No), the common element extraction unit 800 performs a common element extraction process to extract common elements of registered configuration data corresponding to the test case 10221 enabled by the test case extraction unit 108 from the linking information in the linking information storage unit 1023 (step S901).

[0094] Then, the test case extraction unit 108 sets the enabled test case 10221 as a candidate test case, and outputs the candidate test case and the common element to the user terminal 111 via the test case output unit 109 (step S902). After that, the process of step S710 is executed, and the test case extraction process is terminated.

[0095] FIG. 10 is a flowchart illustrating an example of the common element extraction process in step S901 of FIG.

[0096] In the common component extraction process, the common component extraction unit 800 determines whether or not two or more test cases 10221 are enabled (step S1001).

[0097] If two or more test cases 10221 are not enabled (step S1001: No), the common element extraction unit 800 ends the common element extraction process without extracting a common element. In this case, only the candidate test cases are output in step S902 in FIG.

[0098] If two or more test cases 10221 are enabled (step S1001: Yes), the common element extraction unit 800 acquires one of the enabled test cases 10221 (step S1002). For example, the common element extraction unit 800 acquires the oldest test case 10221 among the enabled test cases 10221.

[0099] The common element extraction unit 800 refers to the linking information in the linking information storage unit 1023 to acquire registered configuration data corresponding to the acquired test case 10221 (step S1003). The common element extraction unit 800 acquires one of the items from the acquired registered configuration data as an acquired item (step S1004).

[0100] The common element extraction unit 800 refers to the linking information in the linking information storage unit 1023, and determines whether or not the same item as the obtained item exists in the registered configuration data corresponding to the other test case 10221 (step S1005).

[0101] If the same item exists (step S1005: Yes), the common element extraction unit 800 determines whether or not the item value of the same item is the same as the item value of the obtained item (step S1006).

[0102] If the item values ​​are the same (step S1006: Yes), the common element extraction unit 800 extracts the obtained items as common elements (step S1007). If the items are different (step S1006: No), the common element extraction unit 800 skips the process of step S1007. If the same item does not exist (step S1005: No), the common element extraction unit 800 skips the processes of steps S1006 and S1007.

[0103] Then, the common element extraction unit 800 determines whether or not there is an item that has not yet been acquired in the acquired registered configuration data (step S1008).

[0104] If there is an item that has not been acquired (step S1008: Yes), the common element extraction unit 800 acquires any one of the items that has not been acquired from the acquired registered configuration data (step S1009), and returns to the processing of step S1005.

[0105] If there is no item that has not been acquired (step S1008: No), the common element extraction unit 800 ends the common element extraction process.

[0106] FIG. 11 is a diagram showing an example of a test case candidate display screen that displays candidate test cases and common elements.

[0107] 11(a) includes a test case name 1101 indicating a candidate test case, a configuration data name 1102 indicating a registered configuration data corresponding to the candidate test case, and a common element name 1103 indicating a common element. Here, information on test cases C and D selected as candidate test cases is displayed.

[0108] Similar to the test case candidate display screen 1100a shown in Fig. 11(a), the test case candidate display screen 1100b shown in Fig. 11(b) includes a test case name 1101, a configuration data name 1102, and a common element name 1103. However, the test case candidate display screen 1100b differs from the test case candidate display screen 1100a in that there are two common elements.

[0109] In addition, by considering items with an item similarity lower than 100% as common elements at the time of the determination in S1006, items including minor differences in characters in the item values, such as version differences, can also be extracted as common elements.

[0110] As described above, according to the embodiment, it is possible to present common elements to the user as reasons for selecting candidate test cases, so that the user can grasp elements related to the test cases. Therefore, it is possible to support the user in selecting an appropriate implementation test case from the candidate test cases. EXAMPLES

[0111] In this embodiment, an example is described in which a specified item designated by the user is set as a setting element, and test cases linked to registered configuration data that does not contain that setting element are removed from the candidate test cases, thereby narrowing down the candidate test cases.

[0112] Fig. 12 is a diagram showing a functional configuration of an exploratory testing support device according to a third embodiment of the present disclosure. The exploratory testing support device 100 shown in Fig. 12 is different from the exploratory testing support device 100 according to the second embodiment shown in Fig. 8 in that it further includes an element setting unit 1100.

[0113] The element setting unit 1100 receives setting elements, which are items designated by the user, via the user input unit 103. The element setting unit 1100 sets the setting elements in the storage unit 102, for example.

[0114] 13 is a flowchart for explaining a test case extraction process including an element setting process for setting elements. In the following, the parts different from the test case extraction process of the second embodiment explained with reference to FIG. 9 will be mainly explained.

[0115] First, when the CI tool 112 inputs the system configuration data to the exploratory testing support device 100, it notifies the user terminal 111 of input end information indicating that the input of the system configuration data has ended (step S1301). The input end information may include, for example, a configuration data name indicating the input configuration data, which is the system configuration data input to the exploratory testing support device 100.

[0116] When the user terminal 111 receives the input end information, it displays an input screen for element setting, and then receives from the user designated items to be set in the element setting unit 1200 as setting elements, and inputs the setting elements to the exploratory testing support device 100. When the user input unit 103 of the exploratory testing support device 100 receives the setting elements, the element setting unit 1200 sets the setting elements in the storage unit 102 (step S1302).

[0117] After that, the processes of steps S701 to S708 and S901 are executed. Then, when the process of step S901 is completed, the common element extraction unit 800 performs a filtering process to remove the test case 10221 corresponding to the registered configuration data according to the setting element set by the element setting unit 1200 from the enabled test cases 10221 (step S1303).

[0118] Then, the test case extraction unit 108 sets the test case 10221 that has been subjected to the filtering process as a candidate test case, and outputs the candidate test case and the common element to the user terminal 111 via the test case output unit 109 (step S1304). After that, the process of step S710 is executed, and the test case extraction process is terminated.

[0119] FIG. 14 is a diagram for explaining an example of the filtering process in step S1303 of FIG.

[0120] In the filtering process, the test case extraction unit 108 determines whether or not the setting element set in the element setting unit 1200 is included in the common element extracted by the common element extraction unit 800 (step S1401).

[0121] If the setting element is included in the common elements (step S1401: Yes), the test case extraction unit 108 ends the filtering process. On the other hand, if the setting element is not included in the common elements (step S1401: No), the test case extraction unit 108 acquires one of the enabled test cases 10221 (step S1402).

[0122] The test case extraction unit 108 refers to the linking information in the linking information storage unit 1023, and acquires one of the system configuration definition files of the registered configuration data corresponding to the acquired test case 10221 (step S1403).

[0123] The test case extraction unit 108 determines whether the setting element is included in the acquired system configuration definition file (step S1404).

[0124] If the setting element is not included (step S1404: No), the test case extraction unit 108 invalidates the acquired test case 10221 (step S1405). If the setting element is included (step S1404: Yes), the test case extraction unit 108 skips the process of step S1405.

[0125] Next, the test case extraction unit 108 judges whether or not there is a system configuration definition file that has not been acquired in the registered configuration data corresponding to the acquired test case 10221 (step S1406).

[0126] If there is a system configuration definition file that has not been acquired (step S1406: Yes), the test case extraction unit 108 acquires one of the system configuration definition files that has not yet been acquired from the registered configuration data corresponding to the acquired test case 10221 (step S1407), and returns to the processing of step S1404.

[0127] If there is no system configuration definition file that has not been acquired (step S1406: No), the test case extraction unit 108 determines whether there is any test case 10221 that has not yet been acquired among the enabled test cases 10221 (step S1408).

[0128] If there are test cases 10221 that have not been acquired (step S1408: Yes), the test case extraction unit 108 acquires one of the enabled test cases 10221 that has not yet been acquired (step S1409) and returns to the processing of step S1403.

[0129] If there is no unacquired test case 10221 (step S1408: No), the test case extraction unit 108 ends the filtering process.

[0130] Fig. 15 is a diagram showing an example of an input screen for a user to input a setting element. An input screen 1500 shown in Fig. 15(a) is an example of an input screen before a target element is input, and an input screen 1500 shown in Fig. 15(b) is an example of an input screen after the target element is input.

[0131] The input screen 1500 includes a configuration data name 1501 indicating the system configuration data input by the CI tool 112 to the exploratory testing support device 100, and an input form 1502 for inputting setting elements. The configuration data name 1501 is set to a system configuration definition file group-X.

[0132] As described above, according to this embodiment, test cases associated with registered configuration data that does not include configuration elements are excluded from candidate test cases. Therefore, it is possible to narrow down candidate test cases to test cases related to configuration items, making it easier for users to select test cases to be implemented. EXAMPLES

[0133] In this embodiment, a hardware configuration of the exploratory testing support device 100 described in the first to third embodiments will be described.

[0134] Fig. 16 is a diagram showing an example of a hardware configuration of the exploratory test support device 100. As shown in Fig. 16, the exploratory test support device 100 has a computer system 1600, an input device 1610, and a display device 1620. The computer system 1600 is connected to the input device 1610 and the display device 1620 so as to be able to communicate with each other.

[0135] The computer system 1600 includes a read only memory (ROM) 1601, a central processing unit (CPU) 1602, a random access memory (RAM) 1603, and a storage medium 1604.

[0136] ROM 1601 records various information such as computer programs that define the operation of CPU 1602. Fig. 16 shows, as examples of computer programs, a similarity calculation program 16011 for implementing similarity calculation unit 106, a score calculation program 16012 for implementing score calculation unit 107, a data update program 16013 for implementing weight setting unit 104, threshold setting unit 105, stored data update unit 110, etc., and a test case extraction program 16014 for implementing test case extraction unit 108, but other computer programs may also be present.

[0137] The CPU 1602 reads the program recorded in the ROM 1601 into the RAM 1603 and executes the program using the RAM 1603, thereby realizing the functional configuration of the exploratory testing support device 100 shown in FIG. 1, FIG. 8 or FIG. 12.

[0138] The storage medium 1604 is a large-capacity recording medium that records data in a writable and readable manner, and stores, for example, a test case group 16041 that is a collection of test cases 10221. Note that the storage medium 1604 may be provided outside the computer system 1600.

[0139] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to easily explain the present invention, and is not necessarily limited to those having all the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. In addition, the steps in the processing of each embodiment may be executed in an appropriate order as long as the same result can be obtained.

[0140] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be stored in a storage unit such as a memory, a hard disk, or an SSD (Solid State Drive), or in an IC card, a semiconductor recording medium, a magnetic recording medium, or an optical recording medium.

[0141] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0142] 100: Exploratory test support device 100 101: System configuration data input unit 102: Memory unit 103: User input unit 104: Weight setting unit 105: Threshold setting unit 106: Similarity calculation unit 107: Score calculation unit 108: Test case extraction unit 109: Test case output unit 110: Stored data update unit 111: User terminal 112: CI tool 113: Repository 114: CD tool 115: Application execution environment 116: Exploratory test tool 800: Common element extraction unit 1021: System configuration data storage unit 1022: Test case storage unit 1023: Information storage unit 1100: Element setting unit

Claims

1. a storage unit that stores a plurality of registered configuration data related to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; a similarity calculation unit that calculates data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; an extracting unit that extracts candidate test cases from the storage unit based on each data similarity, the candidate test cases being candidates for test cases when testing the target system; an output unit for outputting the candidate test cases; the similarity calculation unit calculates, for each item of the input configuration data, an item similarity between an item value of the input configuration data and an item value of an item identical to the input item in the registered configuration data, as the data similarity; The extraction unit is a score calculation unit that calculates a score, which is a weighted sum of each item similarity between the registered configuration data and the input configuration data, for each test case associated with the registered configuration data; a test case extraction unit that extracts the candidate test cases based on the scores, the test case extraction unit extracts test cases whose scores are equal to or greater than a threshold as the candidate test cases; The threshold is set for each of the test cases; an input unit that receives a selection result of selecting one of the output candidate test cases; and an update unit that updates at least one of the weighting values ​​of the weighted sums and the threshold value based on a result of the selection.

2. A storage unit that stores a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; a similarity calculation unit that calculates data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; an extracting unit that extracts candidate test cases from the storage unit based on each data similarity, the candidate test cases being candidates for test cases when testing the target system; an output unit for outputting the candidate test cases; the similarity calculation unit calculates, for each item of the input configuration data, an item similarity between an item value of the input configuration data and an item value of an item identical to the input item in the registered configuration data, as the data similarity; The extraction unit is a score calculation unit that calculates a score, which is a weighted sum of each item similarity between the registered configuration data and the input configuration data, for each test case associated with the registered configuration data; a test case extraction unit that extracts the candidate test cases based on the scores, the test case extraction unit extracts test cases whose scores are equal to or greater than a threshold as the candidate test cases; an input unit that receives a selection result of selecting one of the output candidate test cases; and an update unit that updates at least one of the weighting values ​​of the weighted sums and the threshold value based on a result of the selection.

3. A storage unit that stores a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; a similarity calculation unit that calculates data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; an extracting unit that extracts candidate test cases from the storage unit based on each data similarity, the candidate test cases being candidates for test cases when testing the target system; an output unit for outputting the candidate test cases; a common element extraction unit that extracts common elements from among the items of the registered configuration data, the common elements being items whose item similarity to the item values ​​of the items of the input configuration data is equal to or greater than a predetermined value; The output unit further outputs the common element.

4. A storage unit that stores a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; a similarity calculation unit that calculates data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; an extracting unit that extracts candidate test cases from the storage unit based on each data similarity, the candidate test cases being candidates for test cases when testing the target system; an output unit for outputting the candidate test cases; The extraction unit removes test cases linked to the registered configuration data that does not include a specified item from the candidate test cases.

5. A test support method using a test support device, storing each of a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; Calculating data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; Extracting candidate test cases that are candidates for test cases when testing the target system based on each data similarity; Outputting the candidate test cases; In the calculation of the data similarity, for each item of the input configuration data, an item similarity between an item value of the input configuration data and an item value of the same item as the input configuration data in the registration configuration data is calculated as the data similarity; In the extraction of the candidate test cases, Calculating a score for each test case associated with the registered configuration data, which is a weighted sum of the similarities between the registered configuration data and the input configuration data; extracting test cases whose scores are equal to or greater than a threshold as the candidate test cases; The threshold is set for each of the test cases; A selection result is received from one of the output candidate test cases, A test support method, comprising: updating at least one of the weighting values ​​of the weighted sums and the threshold value based on the selection result.

6. A test support method using a test support device, comprising: storing each of a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; Calculating data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; Extracting candidate test cases that are candidates for test cases when testing the target system based on each data similarity; Outputting the candidate test cases; In the calculation of the data similarity, for each item of the input configuration data, an item similarity between an item value of the item and an item value of the same item in the registered configuration data is calculated as the data similarity; In the extraction of the candidate test cases, Calculating a score for each test case associated with the registered configuration data, which is a weighted sum of the similarities between the registered configuration data and the input configuration data; extracting test cases whose scores are equal to or greater than a threshold as the candidate test cases; A selection result is received from one of the output candidate test cases, A test support method, comprising: updating at least one of the weighting values ​​of the weighted sums and the threshold value based on the selection result.

7. A test support method using a test support device, comprising: storing each of a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; Calculating data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; Extracting candidate test cases that are candidates for test cases when testing the target system based on each data similarity; Outputting the candidate test cases; extracting common elements from among the items of the registered configuration data, the common elements being items whose item similarity to the item values ​​of the items of the input configuration data is equal to or greater than a predetermined value; The test support method further outputs the common element.

8. A test support method using a test support device, comprising: storing each of a plurality of registered configuration data relating to the configurations of a plurality of user systems in association with a test case when a test of the user system corresponding to the registered configuration data was performed; Calculating data similarity between input configuration data relating to a configuration of a target system, which is a user system to be tested, and each of the plurality of registered configuration data; Extracting candidate test cases that are candidates for test cases when testing the target system based on each data similarity; removing a test case associated with the registered configuration data that does not include a predetermined designated item from the candidate test cases; and outputting candidate test cases other than the test case.

Images