Software verification device, and software verification method

Abstract

To provide a software verification device and a software verification method for widely covering software specifications and automatically generating a test case having high code coverage.SOLUTION: A software verification device includes: a specification coverage test case generation unit for generating a test case that covers a combination of factors extracted from specification information of software being a test object; a test execution unit for executing a test by the generated test case to the software to generate code coverage information of the software in the executed test case; and an additional test case generation unit for generating an additional test case for increasing code coverage more than a test case related to the code coverage on the basis of a relation between the facts in the test case generated by the specification coverage test case generation unit or levels of the factors, and the generated code coverage information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a software verification device and a software verification method. [Background technology]

[0002] In order to find software defects and improve its quality, it is necessary to verify the software by performing tests on the software. As the number of software test cases increases, the time and computing resources required to perform the tests increase, so it is desirable to prepare test cases that can efficiently find defects with a small number of cases. However, there is a problem in that when a software developer designs and implements such efficient test cases, it takes a lot of man-hours. To solve this problem, a technology has been disclosed that automatically generates test cases for software verification.

[0003] Non-Patent Document 1 discloses a technique for automatically generating test cases using the Pairwise method (referred to as Covering Array in Non-Patent Document 1) from interface specification information described using OpenAPI, which is one of the specification description languages ​​for APIs (Application Programming Interfaces). Non-Patent Document 1 discloses that the Pairwise method is used to generate test cases that efficiently cover all combinations of WebAPI parameters defined in the interface specification information.

[0004] Patent Document 1 also discloses a technique for extracting branch conditions from an activity diagram in UML (Uniform Modeling Language) and generating test cases that cover all the branches. Patent Document 1 also discloses generating test cases that increase code coverage, which indicates which paths in the source code have been taken, by executing the test cases. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016 / 170937 [Non-patent literature]

[0006] [Non-Patent Document 1] Huayao Wu, Lixin Xu, Changhai Nie, “Combinatorial testing of RESTful APIs”, Proceedings of 44th International Conference on Software Engineering, May 2022, pp.426-437. Summary of the Invention [Problem to be solved by the invention]

[0007] When evaluating software quality, there are two ways to evaluate it: from the perspective of specification coverage, which is the extent to which combinations of software specifications have been covered through testing, and from the perspective of code coverage, which is the percentage of the source code that has been executed.

[0008] The above-mentioned technology has a problem in that it is not possible to automatically generate test cases that efficiently satisfy both the completeness of specifications and the code coverage. For example, the technology disclosed in Non-Patent Document 1 does not take code coverage into consideration when generating test cases, and it is not possible to generate test cases that efficiently increase code coverage.

[0009] In addition, the technology disclosed in Patent Document 1 requires the creation of a UML activity diagram that describes the branching conditions within the source code, and this has the problem that it can only be created by a developer who has a deep understanding of the contents of the source code.

[0010] The present invention has been made in consideration of the above circumstances, and has as its object to provide a software verification device and a software verification method that are capable of automatically generating test cases that widely cover software specifications and have high code coverage. [Means for solving the problem]

[0011] One of the present inventions for solving the above problems is a software verification device having a processor and a memory, and including: a specification coverage test case generation unit that has a processor and a memory, and generates test cases that cover combinations of factors extracted from specification information of the software to be tested; a test execution unit that executes tests using the generated test cases on the software and generates code coverage information of the software for the executed test cases; and an additional test case generation unit that generates additional test cases that increase code coverage more than the test cases related to the code coverage information, based on the relationship between the factors or levels of the factors in the test cases generated by the specification coverage test case generation unit and the generated code coverage information. Effect of the Invention

[0012] According to the present invention, test cases that widely cover software specifications and have high code coverage can be automatically generated. Configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of a configuration of a software verification device according to a first exemplary embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram illustrating an example of interface specification information. [Diagram 3] FIG. 13 illustrates an example of a test case; [Figure 4] FIG. 11 illustrates an example of code coverage information. [Diagram 5]FIG. 2 is a diagram illustrating a flow of processing performed by the software verification device according to the first embodiment of the present invention. [Figure 6] FIG. 11 is a diagram illustrating an example of a functional configuration of a software verification device according to a second exemplary embodiment of the present invention. [Figure 7] FIG. 13 is a diagram illustrating an example of Pairwise generation parameters. [Figure 8] FIG. 2 is a diagram illustrating an overview of a process performed by the software verification device. [Figure 9] FIG. 2 is a diagram illustrating an overview of a process performed by the software verification device. [Figure 10] 11 is a process flow diagram illustrating an example of a parameter update process executed by a coverage learning unit. FIG. [Figure 11] FIG. 11 is a process flow diagram illustrating another example of the parameter update process executed by the coverage learning unit. [Figure 12] FIG. 13 is a diagram showing an example of a test result display screen generated by a test result display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications are made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. In the following explanation, various information may be explained using expressions such as "table," "list," and "queue," but the various information may be expressed in other data structures. To indicate independence of data structure, "XX table," "XX list," and the like may be referred to as "XX information." When explaining identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable. When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts. However, when there is no need to distinguish between these multiple components, the subscripts may be omitted. In addition, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., CPU, GPU) to perform a defined process using storage resources (e.g., memory) and / or interface devices (e.g., communication ports) as appropriate, so the subject of the processing may be the processor. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be a calculation unit, and may include a dedicated circuit (e.g., FPGA or ASIC) that performs a specific process. A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing a program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0015] [First embodiment] First, a software verification device and a software verification method according to a first embodiment of the present invention will be described.

[0016] FIG. 1 is a diagram showing a configuration example of a software verification device 100 according to a first embodiment of the present invention. The software verification device 100 is realized by an information processing device including one or more computers. In addition to a configuration example in which the software verification device 100 implements a user interface in itself, the software verification device 100 may use a user terminal connected via an appropriate network as the user interface. In this case, the user terminal is a terminal device operated by a software developer, tester, or the like. Specifically, this user terminal is a personal computer, a smartphone, a tablet terminal, or the like.

[0017] The software verification device 100 is an information processing device arranged in a test environment for software to be verified by testing (hereinafter referred to as "target software"). The software verification device 100 automatically generates efficient test cases required for testing the target software, and outputs the results of tests executed based on the generated test cases.

[0018] As shown in the figure, the software verification device 100 includes a storage device 200, an arithmetic device 250, a memory 260, and an input / output device 270. The storage device 200, the arithmetic device 250, the memory 260, and the input / output device 270 are communicatively connected to each other via a bus.

[0019] The arithmetic unit 250 reads the program 210 stored in the storage device 200 into the memory 260. The CPU (Central Processing Unit) executes various functions, such as reading and writing data, to provide overall control of the entire device, as well as perform various judgments, calculations, and control processes. The memory 260 is composed of a volatile storage element such as a random access memory (RAM) or a read only memory (ROM).

[0020] The input / output device 270 is a device that receives key input or voice input from the user and displays processing data. For example, the input / output device 270 is composed of input devices such as a keyboard, a mouse, a touch panel, a microphone, etc., and a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display, or a print output device such as a printer. The input / output device 270 may also include a network interface card that is connected to an appropriate network and handles communication processing with a user terminal.

[0021] The storage device 200 is composed of an appropriate non-volatile storage element such as an SSD (Solid State Drive), a hard disk drive, etc. In addition to a program 210 for implementing each function required for the software verification device 100 of this embodiment, the storage device 200 stores at least a source code 11 of the target software, interface specification information 12, a test case 111 (to be described later), test result information 121 (to be described later), code coverage information 122 (to be described later), and an additional test case 141 (to be described later).

[0022] Inputs to the software verification device 100 include source code 11 of the target software and interface specification information 12. The interface specification information 12 is specification information that includes information required to generate factors and levels (factor values) used in testing the target software. Factors are the types of parameters and elements to be tested for the software. Furthermore, levels are values ​​or representative values ​​that are permitted to be input to the factors.

[0023] 2 is a diagram showing an example of interface specification information 12. The interface specification information 12 shown in the diagram is information describing each parameter ("ParamA" and "ParamB") that is a factor, the type of each parameter, the range of values ​​that each parameter can take, the number of parameters, etc. For example, the parameter "ParamA" is an int type and has a value of 0 or more.

[0024] For example, in the case of a web service, a specification description based on the OpenAPI specification description can be used as the interface specification information 12. As a method for generating factors and levels from the OpenAPI specification description, the method disclosed in Non-Patent Document 1 can be used.

[0025] Next, as shown in FIG. 1, the program 210 includes programs for implementing the specification coverage test case generation unit 110, the test execution unit 120, the test result display unit 130, and the additional test case generation unit 140, respectively.

[0026] The specification coverage test case generation unit 110 executes a specification coverage test case generation process for generating test cases 111 that cover the specifications of the target software based on the interface specification information 12.

[0027] 3 is a diagram showing an example of the test case 111. In the test case 111 shown in the diagram, the level "0" is set for the factor "ParamA" and the level "a" is set for the factor "ParamB".

[0028] The test execution unit 120 executes the source code 11 of the verification software and the specification coverage test cases. A test execution process is executed to execute a test on the target software using the test cases 111 generated by the generation unit 110. When executing a test, the test execution unit 120 also acquires code coverage indicating which paths in the source code 11 were executed by the execution of the test cases. As a result, the test execution unit 120 outputs test result information 121 including information on whether the test cases were successful or not, and information including code coverage information 122 for the test cases.

[0029] The test result information 121 includes information on whether each test case 111 was successful (OK) or failed (NG), the number of successful test cases 111, and the number of failed test cases 111.

[0030] 4 is a diagram showing an example of the code coverage information 122. The code coverage information 122 includes information on the code coverage of each file in the source code 11 and each function included in each file in each test case 111. The code coverage information 122 shown in the figure includes a code coverage of "70%" for file A, a code coverage of "50%" for file B, and a code coverage of "90%" for file C. Code coverage is the percentage of the source code 11 that has been tested. The method for measuring code coverage may be any method, such as instruction coverage, branch coverage, condition coverage, complex condition coverage, or path combination coverage.

[0031] The test result display unit 130 displays the contents including the test result information 121 and the code coverage information 122 output by the test execution unit 120 on a screen so that a user such as a developer can check them. In addition, if the user uses a project management tool for managing the development status of the project, the test result display unit 130 may transmit information to the project management tool.

[0032] The additional test case generation unit 140 executes an additional test case generation process to generate an additional test case 141 that increases code coverage more than the test case 111 related to the code coverage information 122 by identifying factors or levels that contribute to an increase in code coverage based on the relationship between the factors or levels of the factors in the test case 111 and the code coverage information 122 obtained using the test case 111.

[0033] In addition to a configuration example in which each of the constituent functions of the software verification device 100 is arranged in a single information processing device, the functions may be distributed and arranged in multiple information processing devices connected via a network, or in computing resources on the cloud.

[0034] 5 is a diagram for explaining the flow of processing performed by the software verification device 100 in the first embodiment of the present invention. As shown in the figure, first, the specification coverage test case generation unit 110 generates test cases 111 that cover the specifications of the target software based on the interface specification information 12 (S11). Note that this processing is started, for example, when a predetermined input is made to the software verification device 100 by the user, or at a predetermined timing (for example, a predetermined time, a predetermined time interval), etc. For example, the specification coverage test case generation unit 110 may start this processing when the user specifies the interface specification information 12 of the target software and causes the software verification device 100 to read it.

[0035] For example, the specification coverage test case generation unit 110 may generate test cases that cover all combinations of factors and levels defined in the interface specification information 12, or may generate test cases by a method of randomly generating values ​​for each factor. In addition, as a combination test technique for generating effective combination test cases, a method using an orthogonal array or a method using the Pairwise method (All-Pair method) described later is available. Methods such as the specification coverage test case generation unit 110 may generate test cases using these methods.

[0036] Next, the test execution unit 120 acquires the source code 11 of the target software (S12). Then, the test execution unit 120 executes a test on the target software using the test cases 111 generated in S11 and the source code 11 of the verification software acquired in S12 (S13). For example, the test execution unit 120 sets combinations of factors and levels in the test cases 111 as initial values ​​for the source code 11, and executes the source code 11 with these initial values ​​set.

[0037] When executing a test, the test execution unit 120 also acquires the code coverage of the test case 111 and outputs the code coverage information 122 together with the test result information 121.

[0038] One method of acquiring code coverage in the test execution unit 120 is to execute a test after performing a program conversion on the source code 11, such as inserting a code for outputting information on whether or not each line in the source code 11 has been executed to a log. Alternatively, the test execution unit 120 may acquire code coverage by using a calculation device 250 having a function for acquiring information on the execution position of a program.

[0039] Next, the test result display unit 130 displays on the screen the contents including the test result information 121 and the code coverage information 122 output by the test execution unit 120 (S14).

[0040] On the other hand, the additional test case generation unit 140 generates an additional test case 141 that further increases the code coverage based on the test case 111 and the code coverage information 122 obtained by using the test case 111 (S15). The additional test case generation unit 140 may generate an additional test case 141 that includes, as a test target, a level of a selected factor that contributes to an increase in code coverage and a combination of other factors and the levels of the other factors, in addition to a method of changing a generation parameter of the Pairwise method described later. Alternatively, the additional test case generation unit 140 may identify factors that execute paths that are not executed and their levels based on the code coverage information 122 indicating which paths in the source code 11 have been executed, and generate an additional test case 141 that includes the identified factors and their levels as test targets.

[0041] Alternatively, the additional test case generation unit 140 may generate the additional test cases 141 using a trained model. For example, the additional test case generation unit 140 performs machine learning to generate a model in which the input value is a factor and a level, and the output value is a code coverage. The trained model is constructed based on, for example, an algorithm of a neural network, a decision tree, a random forest, or a support vector machine (SVM).

[0042] When the additional test case generation unit 140 generates the additional test case 141, the test execution unit 120 executes a test on the generated additional test case 141, and generates test result information 121 and code coverage information 122. The test result display unit 130 also displays the contents of the execution result of the test on the additional test case 141.

[0043] In addition, it is also possible to generate a further additional test case 141 in the additional test case generating unit 140 using the additional test case 141 and the code coverage information 122 generated for the additional test case 141. The generation of the additional test case 141 and the test execution for the generated additional test case 141 are carried out in a manner that achieves a desired code coverage. The software verification apparatus 100 may repeat the generation and execution of the additional test cases 141 until a specified test execution time is reached.

[0044] As described above, the software verification device 100 of this embodiment includes: a specification coverage test case generation unit 110 that generates test cases 111 that cover all combinations of factors extracted from interface specification information 12 of the software to be tested; a test execution unit 120 that executes tests using the generated test cases 111 on the software and generates code coverage information 122 of the software in the executed test cases 111; and an additional test case generation unit 140 that generates additional test cases that increase code coverage more than the test cases 111 related to the code coverage information 122, based on the relationship between the factors in the test cases 111 or the levels of the factors and the generated code coverage information 122.

[0045] That is, the software verification device 100 of this embodiment can acquire code coverage information 122 based on test cases that cover all combinations of factors according to the interface specifications, and generate additional test cases that increase code coverage based on the acquired code coverage information 122 and the factors or levels of the test cases 111. This makes it possible to automatically generate test cases that widely cover the software specifications and have high code coverage.

[0046] [Second embodiment] Next, a software verification device and a software verification method according to a second embodiment of the present invention will be described.

[0047] The configuration of the software verification device 100 in this embodiment is the same as that of the software verification device 100 in the first embodiment, and therefore the description thereof will be omitted. Also, in the software verification device 100 in this embodiment, as in the first embodiment, the arithmetic device 250 loads the program 210 stored in the storage device 200 in advance into the memory 260 and executes it, thereby realizing the respective functions of the Pairwise test case generation unit 115, the test execution unit 120, the test result display unit 130, and the coverage learning unit 150, which will be described later. Note that the Pairwise test case generation unit 115 corresponds to the specification coverage test case generation unit 110, and the Pairwise test case generation unit 115 and the coverage learning unit 150 correspond to the additional test case generation unit 140.

[0048] 6 is a diagram showing an example of a functional configuration of the software verification device 100 according to the second embodiment of the present invention. In this diagram, the same components as those in the software verification device 100 of the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0049] As shown in the figure, the software verification device 100 is composed of a pairwise test case generation unit 115, a test execution unit 120, a test result display unit 130, and a coverage learning unit 150. As in the first embodiment, the input to the software verification device 100 includes source code 11 of the target software and interface specification information 12.

[0050] The Pairwise test case generation unit 115 generates test cases 111 by the Pairwise method using information on factors and levels obtained from the interface specification information 12 and Pairwise generation parameters 151 .

[0051] The Pairwise method generates n-factor coverage test cases for a given coverage level n. n-factor coverage means that the test cases generated will test at least once for each level of the n factors included in any n-factor combination of the factors to be verified. For example, one-factor coverage means that the test cases to be executed will test at least once for each level of each factor to be tested. Similarly, two-factor coverage means that the test cases to be executed will test at least once for each level of each of the two factors included in a two-factor combination of the factors to be verified.

[0052] In the Pairwise method, the coverage n can be specified as a common n for all factors, or a different coverage can be specified for each factor, thereby increasing or decreasing the coverage when a specific factor is included. As a specific means for generating test cases using the Pairwise method, the method shown in Japanese Patent No. 2882687 can be used. For example, the Pairwise test case generation unit 115 generates test cases by a method of listing test cases for each relationship between factors and combining test cases for different relationships to generate a single test case.

[0053] 7 is a diagram showing an example of the Pairwise generation parameters 151. The Pairwise generation parameters 151 include information on the coverage and number of levels of each factor. In the Pairwise generation parameters 151 shown in the diagram, the number of levels "4" and the coverage "2" are set for each of the parameters (ParamA, ParamB, ParamC, ParamD) which are factors.

[0054] As a method for setting the initial values ​​of the Pairwise generation parameters 151, a method in which a certain degree of coverage and the number of levels are set in advance for all factors, or a method in which the user individually specifies them may be used.

[0055] Based on the relationship between the test case 111 and the code coverage information 122 obtained using the test case 111, the coverage learning unit 150 updates the value of the pairwise generation parameter 151 so as to increase the code coverage compared to the test case 111 related to the code coverage information 122. The details of the processing by the coverage learning unit 150 will be described later.

[0056] When the Pairwise generation parameters 151 are updated, the Pairwise test case generation unit 115 adds a test case 111 reflecting the updated Pairwise generation parameters 151. The test execution unit 120 executes a test on the added test case 111 and generates test result information 121 and code coverage information 122.

[0057] It is also possible to further update the Pairwise generation parameters 151 in the coverage learning unit 150 by using the added test case 111 and the code coverage information 122 for the added test case 111. Addition of the test case 111, test execution of the generated test case 111, and update of the Pairwise generation parameters 151 can be repeated until a desired code coverage is obtained. Furthermore, the software verification device 100 may repeat the addition and execution of the test case 111 until a specified test execution time is reached.

[0058] <Processing Overview> 8 and 9 are diagrams for explaining an outline of the process performed in the software verification device 100. For example, the software verification device 100 performs a process when a user inputs a predetermined value to the software verification device 100, or when a predetermined timing (for example, a predetermined time, a predetermined time interval, etc.) is reached. For example, the software verification device 100 may execute the process shown in the figure when the user specifies the source code 11 and the interface specification information 12 of the target software and causes the software verification device 100 to read them.

[0059] First, the Pairwise test case generation unit 115 of the software verification device 100 extracts factors to be tested and the levels of each factor from the interface specification information 12 (S90 in FIG. 9). In the example shown in FIG. 8, the Pairwise test case generation unit 115 extracts the factors "paramA" and "paramB" from the interface specification information 12, extracts the levels "0", "50", and "100" of the factor "paramA", and extracts the levels "'', "'a'", and "'bc'" of the factor "paramB".

[0060] Next, the Pairwise test case generation unit 115 generates test cases 111 for factor coverage by the Pairwise method using the Pairwise generation parameters 151 (S100 in FIG. 9). In the example shown in FIG. 8, the test cases 111 include a test case in which the factor "paramA" is set to level "0" and the factor "paramB" is set to level "'a'".

[0061] Next, the test execution unit 120 executes tests on the generated test cases 111, and generates test result information 121 and code coverage information 122 (S200 in FIG. 9). In the example shown in FIG. 8, the code coverage information 122 indicates that the code coverage of file A is "70%", the code coverage of file B is "50%", and the code coverage of file C is "90%".

[0062] Next, the test result display unit 130 displays information indicating the test execution results, including the generated test result information 121 and the code coverage information 122, on the screen (S300 in FIG. 9).

[0063] Next, the coverage learning unit 150 determines whether or not the test termination condition is satisfied (S400 in FIG. 9). For example, the coverage learning unit 150 may refer to the code coverage information 122 and determine that the termination condition is satisfied when the code coverage of all files or functions is equal to or greater than a preset threshold (for example, 60%). The threshold to be compared with the code coverage is preset by a user. Alternatively, the coverage learning unit 150 may determine that the termination condition is satisfied when a preset test execution time has elapsed since the first test was executed. The test execution time is preset by a user. Alternatively, the coverage learning unit 150 may determine that the termination condition is satisfied when the number of updates of the Pairwise generation parameters 151 is equal to or greater than a preset threshold. The threshold to be compared with the number of updates is preset by a user. Alternatively, the coverage learning unit 150 may determine that the termination condition is satisfied when a predetermined input is received from a user.

[0064] When it is determined that the termination condition is not satisfied (S400: No in FIG. 9), the coverage learning unit 150 executes a parameter update process for updating the value of the Pairwise generation parameters 151 so as to increase the code coverage, using the relationship between the test case 111 and the code coverage information 122 obtained using the test case 111 (S500 in FIG. 9). For example, in the example shown in FIG. 8, if the code coverage of file B with low code coverage does not change even if the level of the factor "paramA" is changed, and the code coverage of file B increases when the level of the factor "paramB" is changed, the coverage learning unit 150 adds a level to the factor "paramB" in the Pairwise generation parameters 151. After that, the software verification device 100 returns to the process of S100 in FIG. 9.

[0065] On the other hand, if coverage learning section 150 determines that the end condition is satisfied (S400 in FIG. 9: Yes), it ends this process.

[0066] By updating the values ​​of the Pairwise generation parameters 151 in this manner, it is possible to increase code coverage in testing. Next, the parameter update process executed by the coverage learning unit 150 will be described in detail.

[0067] <First parameter update process> 10 is a process flow diagram for explaining an example of parameter update processing S500 executed by the coverage learning unit 150. The process example shown in this figure shows an example of a method for updating the value of the number of levels in the Pairwise generation parameters 151 by using the test cases 111 and the code coverage information 122. In this example, the coverage learning unit 150 generates additional test cases by identifying one or more factors that contribute to an increase in code coverage and increasing the number of levels for the identified factors.

[0068] First, the coverage learning unit 150 determines a file or function (hereinafter, referred to as a "target file or target function") for which code coverage is to be increased (S501). The method of determining the target file or target function includes, for example, a method in which a user such as a developer or tester designates the file or function, or a method in which a file or function that does not satisfy a preset code coverage standard is automatically selected and determined.

[0069] Next, the coverage learning unit 150 selects one of the factors extracted from the interface specification information 12 (S502).

[0070] Next, the coverage learning unit 150 compares the change (difference) in code coverage of the target file or target function between test cases that set different levels for the selected factor. Then, the coverage learning unit 150 calculates the sum of the change in code coverage for all combinations of test cases that set different levels (S503).

[0071] Next, the coverage learning section 150 determines whether or not there are any remaining factors for which the amount of change in code coverage has not been calculated (factors for which the process of S503 has not been executed) (S504).

[0072] If there are remaining factors (S504: Yes), the coverage learning unit 150 selects another factor and repeats the processes of S502 and S503 to calculate the amount of change in code coverage for each factor.

[0073] On the other hand, if there are no remaining factors (the change in code coverage for all factors has been calculated) (S504: No), the coverage learning unit 150 selects the factor that maximizes the sum of the change in code coverage calculated for each factor, and increases the number of levels for that factor in the Pairwise generated parameters 151 by 1 (S505). After that, the coverage learning unit 150 ends this first parameter update process.

[0074] For factors that cause a large change in code coverage in a target file or target function when a value is changed, when a test case with a new level for that factor is added, the code coverage rate is expected to increase by having a code coverage different from the existing code coverage. Therefore, in the first parameter update process, the coverage learning unit 150 increases the number of levels for the factor with the largest change in code coverage and generates a test case with the new level set, thereby efficiently increasing the code coverage.

[0075] In the above-described processing example, the coverage learning unit 150 selects one factor for which the number of levels is to be increased, but this is not limiting, and the number of levels of a plurality of factors may be increased. For example, The coverage learning unit 150 may increase the number of levels of a predetermined number of factors in descending order of the amount of change in code coverage, or may increase the number of levels of all factors whose amount of change in code coverage is equal to or greater than a predetermined value. Moreover, the value by which the number of levels is increased is not limited to 1, and may be 2 or more.

[0076] <Second parameter update process> Fig. 11 is a process flow diagram for explaining another example of the parameter update process S510 executed by the coverage learning unit 150. This diagram shows a process example different from the first parameter update process shown in Fig. 10. The process example shown in this diagram shows one method of updating the coverage value in the Pairwise generated parameters 151 by using the test cases 111 and the code coverage information 122. In this example, the coverage learning unit 150 identifies a level of the selected factor at which the code coverage is greater than other levels, and generates an additional test case including a combination of the selected factor and the identified level and a combination of the other factor and the level of the other factor as test targets.

[0077] First, the coverage learning unit 150 determines a target file or a target function for increasing the code coverage (S511). The method for determining the target file or the target function is the same as that of S501 described above, and therefore the description will be omitted.

[0078] Next, the coverage learning unit 150 selects one of the factors defined in the interface specification information 12 (S512). The method of selecting a factor may be a method designated by a user such as a developer or a tester, a method similar to the method of selecting a factor to be updated in the first parameter update process, or a method of random selection.

[0079] Next, the coverage learning unit 150 selects one of the levels of the selected factor (S513). Then, the coverage learning unit 150 calculates the sum of the code coverage of the target files or target functions of the test cases for which the selected level is set (S514).

[0080] Next, the coverage learning section 150 determines whether or not there are any remaining levels (levels for which the process of S514 has not been executed) for which the sum of code coverage has not been calculated for the selected factors (S515).

[0081] If there are remaining levels (S515: Yes), the coverage learning section 150 selects another level and repeats the processes of S513 and S514 to calculate the sum of the code coverage for each level.

[0082] On the other hand, if there are no remaining levels (the sum of the code coverages of all the levels of the selected factor has been calculated) (S515: No), the coverage learning unit 150 selects the level that maximizes the sum of the code coverages calculated for each level. Then, the coverage learning unit 150 increases the coverage in the Pairwise generated parameters 151 by 1 for the factor having the selected level, under the constraint that the level is the selected level (S516). After that, the coverage learning unit 150 ends this second parameter update process.

[0083] For example, in source code where the function to be called changes depending on the value of a factor, when a specific value is specified for that factor, there are many cases where changing the values ​​of other factors does not increase the code coverage of the called function at all. On the other hand, for existing test cases where the code coverage is at a relatively high level, it is expected that the coverage will increase if the values ​​of other factors are changed while maintaining that level. Therefore, in this second parameter update process, the coverage learning unit 150 fixes the level with the largest total amount of code coverage and then increases the coverage, thereby adding test cases having that level and efficiently improving the code coverage. Increase coverage.

[0084] In the above-mentioned processing example, the coverage learning unit 150 increases the coverage of the factor having the level at which the sum of the code coverage is maximized, but this is not limited to the above, and the coverage of a plurality of factors (for example, all factors) may be increased under the constraint that the level at which the sum of the code coverage is maximized is fixed. Also, the value by which the coverage is increased is not limited to 1, and may be 2 or more.

[0085] The first parameter update process shown in Fig. 10 and the second parameter update process shown in Fig. 11 are examples of processes executed by the coverage learning unit 150, and are not limited to these. For example, the process in the coverage learning unit 150 may include both the first parameter update process and the second parameter update process.

[0086] In addition, the processing in the coverage learning unit 150 may determine parameters to be updated using a machine learning algorithm including a random forest, a deep neural network, a support vector machine, or the like.

[0087] (Test result display screen) 12 is a diagram showing an example of a test result display screen 600 generated by the test result display unit 130. The display example shown in this figure displays the results when the number of levels 4 and the coverage level 2 are specified for all factors as the initial Pairwise generation parameters 151 for software having an interface for four types of factors (paramA to paramD).

[0088] The test result display screen 600 includes a basic test case result display area 610 and an additional test case result display area 620 .

[0089] The basic test case result display area 610 is an area that displays the results of tests on basic test cases generated by the Pairwise method using the initial Pairwise generation parameters 151. As shown in the figure, the basic test case result display area 610 includes a parameter display area 611, a test result display area 612, and a code coverage display area 613.

[0090] The parameter display area 611 is an area that displays the number of levels and coverage of each factor (parameter) of the initial Pairwise generated parameters 151. In this display example, the number of levels of all factors is "4", and the coverage is "2".

[0091] The test result display area 612 is an area that displays the number of successful (OK) test cases and the number of failed (NG) test cases as a result of executing a test for each basic test case. In this display example, 50 basic test cases are generated, and the test result display area 612 displays that all the test cases are successful.

[0092] The code coverage display area 613 is an area that displays the code coverage (in "%)" for each file and each function that constitutes the source code 11 of the target software. In this display example, the code coverage target is set to 60%, and the test result display unit 130 highlights in bold the code coverage of files and functions that do not meet the code coverage target in the code coverage display area 613. This allows the user to easily understand the files or functions whose code coverage does not reach the target.

[0093] In this example, the coverage learning unit 150 detects whether the code coverage target is not met. In the case of a method in which the user is made to select the target files or target functions for increasing the code coverage, the coverage learning unit 150 may receive a selection input of the target files or target functions from the user on the test result display screen 600.

[0094] The added test case result display area 620 is an area that displays the results of tests on added test cases newly generated by the Pairwise method using the Pairwise generation parameters 151 updated by the coverage learning unit 150. As shown in the figure, the added test case result display area 620 includes an updated parameter display area 621, an added test result display area 622, and an updated code coverage display area 623.

[0095] The updated parameter display area 621 is an area that displays the number of levels and coverage of each factor (parameter) of the updated Pairwise generation parameters 151. The test result display unit 130 displays the number of levels or coverage that has been changed from the initial Pairwise generation parameters 151 in bold and underlined text. This allows the user to easily check the values ​​of the updated Pairwise generation parameters 151. In this example, the display in the updated parameter display area 621 shows that the number of levels of the factor "paramB" has increased from "4" to "5", and that the coverage of the factor "paramC" has increased from "2" to "3".

[0096] The additional test result display area 622 is an area that displays the number of successful (OK) test cases and the number of failed (NG) test cases as a result of executing each additional test case. In this display example, 30 additional test cases have been added, and the additional test result display area 622 displays that all of the test cases are successful.

[0097] The updated code coverage display area 623 is an area that displays the code coverage for each file and each function that constitutes the source code 11 of the target software, combining the basic test cases and the additional test cases. In this display example, the updated code coverage display area 623 shows that the additional test cases have increased the code coverage of the file "fileA" and the functions in that file, achieving a code coverage of 60% or more.

[0098] The information displayed on the test result display screen 600 is not limited to the above, and may include, for example, code coverage information for each line of a function indicating whether or not it has been executed by a test case.

[0099] Furthermore, the information contained in the test result display screen 600 may be displayed on the screen via the input / output device 270, or may be printed on paper as a report, or may be notified to a developer or other user by sending it as an email via a network, or the like.

[0100] Furthermore, the manner in which each item is highlighted on the test result display screen 600 is not limited to the above. For example, the test result display unit 130 may highlight each item by changing the background color or text color.

[0101] As described above, the software verification device 100 of this embodiment extracts each factor of the software and the level of each factor from the interface specification information 12, and generates test cases 111 of factor coverage testing in which a test is executed at least once for each level of each factor included in the extracted combination of factors.

[0102] With this configuration, it is possible to automatically generate test cases 111 that efficiently cover all possible combinations of values ​​for a combination of multiple factors.

[0103] Furthermore, the software verification device 100 of this embodiment generates additional test cases by identifying factors or levels that contribute to an increase in code coverage based on the generated code coverage information 122.

[0104] With this configuration, additional test cases can be generated based on factors or levels that contribute to an increase in code coverage, and therefore additional test cases that increase code coverage can be generated more efficiently.

[0105] Moreover, the software verification device 100 of this embodiment identifies one or more factors that contribute to an increase in code coverage, and generates additional test cases by increasing the number of levels for the identified factors.

[0106] As described above, for factors that contribute to an increase in code coverage, when test cases with new levels are added, the code coverage is expected to increase. Therefore, by increasing the number of levels of factors that contribute to an increase in code coverage, additional test cases that increase the code coverage can be generated more efficiently.

[0107] Moreover, the software verification device 100 of this embodiment selects a file or function for which code coverage is to be increased, and identifies, among test cases 111 with different levels set, factors that cause a larger change in code coverage for the selected target than other factors as factors that contribute to the increase in code coverage.

[0108] With this configuration, factors that contribute to an increase in the code coverage of the target file or function can be identified with higher accuracy, and additional test cases that increase the code coverage of the target file or function can be efficiently generated.

[0109] Moreover, the software verification device 100 of this embodiment identifies a level of the selected factor at which code coverage is greater than other levels, and generates additional test cases that include, as test targets, a combination of the selected factor and the identified level, and a combination of the other factors and the levels of the other factors.

[0110] As described above, for a relatively high level of code coverage, if the level is fixed and the values ​​of other factors are changed, the code coverage is expected to increase. Therefore, by generating additional test cases including a high level of code coverage, additional test cases that increase the code coverage can be generated more efficiently.

[0111] Moreover, the software verification device 100 of this embodiment selects a file or function for which code coverage is to be increased, identifies a level at which code coverage for the selected target is greater than other levels, and generates additional test cases that include, as test targets, a combination of the selected factor and the identified level, and a combination of other factors and the levels of the other factors.

[0112] With this configuration, it is possible to more accurately identify the level that contributes to increasing the code coverage of the target file or function, and thus to efficiently generate additional test cases that increase the code coverage of the target file or function.

[0113] The present invention is not limited to the above-described embodiment, and can be implemented using any components without departing from the scope of the present invention. The above-described embodiment and modified examples are merely examples, and the present invention is not limited to these contents as long as the characteristics of the invention are not impaired. In addition, although various embodiments and modified examples have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

[0114] For example, part of the hardware included in each device of this embodiment may be provided in another device.

[0115] Furthermore, each program of the software verification device may be provided in another device, a certain program may be made up of a plurality of programs, or a plurality of programs may be integrated into a single program. [Explanation of symbols]

[0116] 11 Source Code 12 Interface specification information 100 Software verification device 110 Specification Coverage Test Case Generation Unit 111 Test Cases 115 Pairwise test case generation unit 120 Test Execution Department 121 Test result information 122 Code Coverage Information 130 Test result display section 140 Additional Test Case Generation Unit 141 additional test cases 150 Coverage Learning Department 151 Pairwise generation parameters 200 Storage device 210 Programs 250 Arithmetic equipment 260 Memory 270 I / O Devices 600 Test result display screen

Claims

1. A processor and a memory, a specification coverage test case generation unit that generates test cases that cover combinations of factors extracted from specification information of the software to be tested; a test execution unit that executes a test using the generated test cases on the software and generates code coverage information for the software for the executed test cases; an additional test case generation unit that generates an additional test case that increases code coverage more than a test case related to the code coverage information based on a relationship between a factor or a level of the factor in the test case generated by the specification coverage test case generation unit and the generated code coverage information; A software verification device comprising:

2. The specification coverage test case generation unit extracting each factor of the software and a level of each factor from the specification information, and generating a test case in which a test is executed at least once for each level of each factor included in the extracted combination of factors; 2. The software verification device according to claim 1.

3. The additional test case generation unit includes: generating additional test cases by identifying factors or levels that contribute to an increase in the code coverage based on the generated code coverage information; 3. The software verification device according to claim 2.

4. The additional test case generation unit includes: Identifying one or more factors that contribute to an increase in code coverage, and generating additional test cases by increasing the variety of levels for the identified factors; 4. The software verification device according to claim 3.

5. The additional test case generation unit includes: Selecting a file or function for which code coverage is to be increased, and identifying factors that cause a larger change in code coverage for the selected file or function than other factors between test cases with different levels as factors that contribute to the increase in code coverage; 5. The software verification device according to claim 4.

6. The additional test case generation unit includes: Identifying a level of the selected factor at which code coverage is greater than other levels, and generating additional test cases including a combination of the selected factor and the identified level, and a combination of the other factor and the level of the other factor as test targets.

4. The software verification device according to claim 3.

7. The additional test case generation unit includes: selecting a file or function for which code coverage is to be increased, identifying a level at which code coverage is greater than other levels for the selected file or function, and generating additional test cases including a combination of the selected factor and the identified level, and a combination of other factors and levels of the other factors as test targets; 7. The software verification device according to claim 6.

8. An information processing device, A specification coverage test case generation process that generates test cases that cover combinations of factors extracted from specification information of the software to be tested; a test execution process for executing a test based on the generated test cases on the software and generating code coverage information of the software for the executed test cases; and generating an additional test case based on a relationship between a factor in the generated test case or a level of the factor and the generated code coverage information, the additional test case being generated to increase the code coverage more than the test case related to the code coverage information. Software verification methods.

9. The information processing device, In the specification coverage test case generation process, extracting each factor of the software and a level of each factor from the specification information, and generating a test case in which a test is executed at least once for each level of each factor included in the extracted combination of factors; The method of claim 8 .

10. The information processing device, In the additional test case generation process, generating additional test cases by identifying factors or levels that contribute to an increase in the code coverage based on the generated code coverage information; The method of claim 9 .

11. The information processing device, In the additional test case generation process, Identifying one or more factors that contribute to an increase in code coverage, and generating additional test cases by increasing the variety of levels for the identified factors; The method of claim 10.

12. The information processing device, In the additional test case generation process, Selecting a file or function for which code coverage is to be increased, and identifying factors that cause a larger change in code coverage for the selected file or function than other factors between test cases with different levels as factors that contribute to the increase in code coverage; The method of claim 11 .

13. The information processing device, In the additional test case generation process, Identifying a level of the selected factor at which code coverage is greater than other levels, and generating additional test cases including a combination of the selected factor and the identified level, and a combination of the other factor and the level of the other factor as test targets. The method of claim 10.

14. The information processing device, In the additional test case generation process, selecting a file or function for which code coverage is to be increased, identifying a level at which code coverage is greater than other levels for the selected file or function, and generating additional test cases including a combination of the selected factor and the identified level, and a combination of other factors and levels of the other factors as test targets; The method of claim 13.

Images