Test generation device, test generation method, and program

Abstract

A test generation device provided with a comparison unit for comparing an output that is from a system to be tested on the basis of a given input and an output from a reference model that is on the basis of a given input, and a test generation unit for saving the input and the output from the system to be tested in a storage unit.

Description

[Technical field]

[0001] The present invention relates to techniques for creating inputs and outputs for use in testing software. [Background technology]

[0002] One method of software testing is to provide input to the software, execute the software, and compare the output (execution result) with the expected result to determine whether the software is normal or not. The expected result is the result when the software is executed correctly.

[0003] To reduce operations, it is desirable to be able to automatically execute software tests. To create tests that can be executed automatically, it is necessary to create inputs and outputs that are expected results for those inputs in advance.

[0004] By preparing such inputs and outputs, it is possible to automatically perform regression testing, for example, by automatically inputting input into upgraded software and determining whether the output matches the expected result. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US2014 / 0115565A1 Summary of the Invention [Problem to be solved by the invention]

[0006] Creating inputs for automated testing is relatively easy, but creating the expected outputs requires a lot of work and is not easy.

[0007] The present invention has been made in consideration of the above-mentioned points, and an object of the present invention is to provide a technique that makes it possible to automatically create inputs and outputs used for testing software. [Means for solving the problem]

[0008] According to the disclosed technology, a comparison unit that compares an output of a system under test executed based on a certain input with an output of a reference model executed based on the input, The comparison section If it is determined that the output of the system under test and the output of the reference model match, The input and the output of the system under test The inputs and outputs are stored in a file for use in later regression testing. A test generator that stores the test results in a storage unit; The test generation unit generates a script that executes the upgraded system under test using the input stored in the storage unit, compares the output of the upgraded system under test with the output stored in the storage unit, and determines whether the two match. A test generator is provided. Effect of the Invention

[0009] The disclosed technology makes it possible to automatically create inputs and outputs used for testing software. [Brief description of the drawings]

[0010] [Figure 1] 1 is an overall configuration diagram of a system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a configuration diagram of a test generation device. [Diagram 3] 13 is a flowchart showing a process flow. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] (Outline of the embodiment) The test method (setup) in this embodiment is a general method used for unit testing, etc. Specifically, first, the code of the software to be tested is created.

[0013] In software development, when multiple programming languages ​​are combined (mix and match), code may be created using a method of converting one language into another. For example, a development method may be used in which code is created in a high-level language and then converted into code in a low-level language (production code). When such a development method is used, at least two versions of the code will exist.

[0014] Next, test inputs are created, either manually or by using an automated test generator. The test inputs are then used to execute tests on the software under test and the reference model. Note that "software" may also be referred to as "system." That is, the software under test may also be referred to as the system under test.

[0015] Next, the execution results of the system under test are compared with those of the reference model, and the inputs and outputs of tests that are determined to match are saved. The saved inputs and outputs can be used for subsequent regression testing. However, testing using the saved inputs and outputs is not limited to regression testing.

[0016] That is, in this embodiment, inputs and outputs for subsequent regression testing can be automatically generated as a byproduct of testing in the code generation process. A set of inputs and outputs may be called a "test." The present embodiment will be described in more detail below.

[0017] (System configuration example) An example of a system configuration according to this embodiment is described with reference to Fig. 1. As shown in Fig. 1, this system includes a test execution device A, a test execution device B, and a test generation device 100. Test execution device A and test execution device B are each connected to test generation device 100 via a network.

[0018] Each of the test execution device A, the test execution device B, and the test generation device 100 is a computer capable of executing software (programs). Each of the test execution device A, the test execution device B, and the test generation device 100 may be a physical machine or a virtual machine on the cloud.

[0019] In this embodiment, three devices are used as the test execution device A, the test execution device B, and the test generation device 100, but this is merely an example. For example, one device including the functions of the test execution device A, the test execution device B, and the test generation device 100 may be used.

[0020] Both test execution device A and test execution device B execute software using test inputs and output the execution results. "Executing software using test inputs" may be called "testing."

[0021] In both test execution device A and test execution device B, input may be made manually to operate the software, or it may be made automatically using a script or the like. The execution results (output) of the tests by test execution device A and test execution device B are input to test generation device 100.

[0022] The execution results that are output may be any information, such as numerical values, a screen image, a table stored in a database, or an execution log.

[0023] Test generation device 100 compares the output from test execution device A with the output from test execution device B to determine whether they match, and depending on the result of the determination, stores the input and output used in the test. Details will be described later.

[0024] In this embodiment, it is assumed that the test execution device A executes a system under test (software), and the test execution device B executes a reference model.

[0025] A reference model is a system that is known to operate correctly (to produce correct output), and by comparing the output of the system under test with the output of the reference model, it is possible to determine whether the output of the system under test is correct or not.

[0026] For example, the reference model is software that operates stably before the upgrade, and the system under test is an upgraded version of software that has the same functions as the reference model. In addition, when developing using mix-and-match, either the code before or after conversion can be used as the reference model software.

[0027] In addition, there are cases where the test execution device B that executes the reference model is a certain model of terminal that operates stably, and the system under test is software that has been modified from the reference model for the test execution device A, which is a different model of terminal from the test execution device B.

[0028] (Example of configuration and operation of test generation device) Fig. 2 shows an example of the configuration of the test generation device 100. As shown in Fig. 2, the test generation device 100 includes an input section 110, a comparison section 120, a test generation section 130, an output section 140, and a storage section 150. Note that the storage section 150 may be a database server connected to an external network of the test generation device 100.

[0029] An example of the operation of a system including the test generation device 100 will be described with reference to the flowchart of FIG.

[0030] In the following example, it is assumed that the system under test and the reference model have already been created and installed on test execution devices A and B so that they can be tested.

[0031] <S1(ステップ1)> In S1, a test input is created. The test input may be, for example, a numerical value to be input to the system, an input procedure to the system (such as pressing button A, inputting value B, clicking OK, etc.), or something else. The test input may be created manually or automatically.

[0032] The test inputs created in S1 are provided to the system under test of test execution device A and to the reference model of test execution device B.

[0033] <s2> In S2, the test execution device A executes the system under test based on the given input. The execution result (output) is input to the input section 110 of the test generation device 100.

[0034] <s3> In S3, the test execution device B executes the reference model based on the given input. The execution result (output) is input to the input section 110 of the test generation device 100.

[0035] <s4> In S4, the comparison section 120 of the test generation device 100 compares the output of the system under test with the output of the reference model to determine whether they match.

[0036] In determining whether or not they match, depending on the type of output data, they may be determined to match even if they do not match exactly. For example, in the case of outputting a person's height in response to a test input, if the system under test outputs 190 cm and the reference model outputs 6 feet 3 inches, the comparison unit 120 determines that they match.

[0037] In such a case, the output from the system under test (190 cm) can be used as the "output" of the test to be generated, making it easier to make comparisons in subsequent regression tests (e.g. tests on a system that is an upgraded version of the current system under test).

[0038] Although the comparison is performed automatically by the comparison unit 120 here, the comparison may be performed manually. For example, the comparison may be performed manually when it is difficult to perform an automatic comparison or when automatic comparison is unable to determine whether the data match or not. When the comparison is performed manually, the person who performed the comparison inputs the comparison result (whether or not the data match) to the comparison unit 120.

[0039] <s5> If the comparison result in S4 is "mismatch," proceed to S5 and debug the system under test.

[0040] <s6> If the comparison result in S4 is a "match," test generation unit 130 stores the pair of inputs for the test that resulted in a "match" and the output for that input in storage unit 150. That is, a "test" consisting of a pair of inputs and outputs is generated.

[0041] In addition, if the output of the system under test and the output of the reference model match exactly (exactly), either the output of the system under test or the output of the reference model may be used as the "output" for test generation.

[0042] As in the case described above, when the comparison unit 120 determines that there is a "match" but not a strict match, the output of the system under test may be used as the "output" for test generation. Also, when the comparison unit 120 determines that there is a "match" but not a strict match, both the output of the system under test and the output of the reference model may be used as the "output" for test generation.

[0043] By performing S1 to S6 for multiple tests, the tests are stored in the memory unit 150 in the form of, for example, "Test 1 = {input 1, output 1}, Test 2 = {input 2, output 2}, Test 3 = {input 3, output 3}, ...."

[0044] <s7> In S7, for example, the test generator 140 creates an automated test (script) using the data (input and output) saved in S6. This script executes the system under test using the input in the data saved in S6, and compares the output with the output in the data saved in S6 to determine whether the two match.

[0045] However, creating such a script and using it for regression testing is just one example. It is also possible to manually perform regression testing using the data (input and output) saved in S6.

[0046] In the above example, if the comparison unit 120 determines in S6 that the output of the system under test and the output of the reference model match, the input and output are stored, but if the comparison unit 120 determines that the output of the system under test and the output of the reference model do not match, the input and the output of the system under test may be stored. By using the stored data in this case in a later regression test, a test can be performed to confirm that the output for the input is different from the output of the stored data as a correct result.

[0047] In addition, when the comparison unit 120 determines that the output of the system under test and the output of the reference model match, the input and the output of the system under test may be stored together with information indicating "match," and when the comparison unit 120 determines that the output of the system under test and the output of the reference model do not match, the input and the output of the system under test may be stored together with information indicating "mismatch."

[0048] (Hardware configuration example) The test generation device 100 can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0049] That is, the test generation device 100 can be realized by executing a program corresponding to the processing performed by the test generation device 100, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0050] Fig. 6 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 6 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus BS.

[0051] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0052] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the test generation device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0053] (Effects of the embodiment) As described above, the technique according to the present embodiment makes it possible to automatically create inputs and outputs used for testing software.

[0054] In particular, the technology according to the present embodiment makes it possible to automatically create, as a by-product of the testing, inputs and outputs suitable for regression testing for maintenance after software development.

[0055] (Summary of the embodiment) This specification discloses at least the following test generation device, test generation method, and program: (Section 1) a comparison unit for comparing an output of the system under test executed based on a certain input with an output of the reference model executed based on the said input; a test generation unit that stores the input and the output of the system under test in a storage unit based on a comparison result by the comparison unit; A test generation device comprising: (Section 2) The test generation unit stores the input and the output of the system under test in the storage unit when the comparison unit determines that the output of the system under test and the output of the reference model match. 2. A test generation apparatus as recited in claim 1. (Section 3) The test generation unit stores the input and the output of the system under test in the storage unit when the comparison unit determines that the output of the system under test and the output of the reference model do not match. 3. A test generation apparatus according to claim 1 or 2. (Section 4) 1. A computer implemented test generation method comprising: a comparison step for comparing an output of the system under test executed on an input with an output of the reference model executed on said input; a test generation step of storing the input and the output of the system under test in a storage unit based on a comparison result of the comparison step; A test generation method comprising: (Section 5) A program for causing a computer to function as a comparison unit and a test generation unit in the test generation device according to any one of claims 1 to 3.

[0056] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0057] A, B Test execution device 100 Test Generation Device 110 Input section 120 Comparison section 130 Test Generation Unit 140 Output section 150 Storage section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device

Claims

1. a comparison unit for comparing an output of the system under test executed based on a certain input with an output of the reference model executed based on the said input; a test generation unit that, when the comparison unit determines that an output of the system under test and an output of the reference model match, stores the input and the output of the system under test in a storage unit for use in a subsequent regression test; The test generation unit generates a script that executes the upgraded system under test using the input stored in the storage unit, compares the output of the upgraded system under test with the output stored in the storage unit, and determines whether the two match. Test generation equipment.

2. The test generation unit stores the input and the output of the system under test in the storage unit when the comparison unit determines that the output of the system under test and the output of the reference model do not match.

2. The test generation apparatus of claim 1.

3. 1. A computer implemented test generation method comprising: a comparison step for comparing an output of the system under test executed on an input with an output of the reference model executed on said input; a test generation step of storing the input and the output of the system under test in a storage unit when it is determined by the comparison step that the output of the system under test and the output of the reference model match, in order to use the input and the output of the system under test in a later regression test; In the test generation step, the computer executes the upgraded system under test using the input stored in the storage unit, and generates a script that compares the output of the system under test with the output stored in the storage unit to determine whether the two match. Test generation methods.

4. 3. A program for causing a computer to function as a comparison section and a test generation section in the test generation device according to claim 1.

Images