Software testing device and software testing method

Abstract

To provide a software testing device and a software testing method.SOLUTION: In a test system, a test device of software in which memory conflict may occur includes a CPU and a storage device including a memory and an external storage device. The CPU includes: a scenario processing unit for executing a test program of a test scenario including a test program of software and assumed results obtained by executing the test program stored in the storage device; a test results processing unit that determines whether or not execution results of the test program match the assumed results and records test result information in the storage device; a conflict check processing unit that detects memory conflict in which a plurality of programs constituting the software write different values in the same address when the test program is executed; and a state table generation processing unit that prepares a state table indicating a test case for each conflicting memory state in which a test is performed based on memory conflict detected by the conflict check processing unit in the storage device.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] In the field of software, there are modules such as threads, processes, and tasks that can be executed in parallel as the smallest execution unit of a program in a microprocessor or operating system. Below, we will explain threads as a representative of these.

[0002] When sharing information between threads, multiple threads read the information into the same memory and share it.

[0003] In this case, when threads operating in parallel write different values ​​to the same memory, this is called a memory conflict, and the value of the memory after the memory conflict occurs is called a conflicted memory state.

[0004] Even with the same input, the conflicting memory state may change depending on the order of thread execution. This can lead to unintended conflicting memory states, which can cause software to behave in an undesigned manner.

[0005] By controlling memory access through exclusive control, it is possible to prevent unintended memory conflicts.

[0006] However, exclusive control causes software delays, so it cannot be used in systems where real-time performance is important, and memory contention may occur in such systems.

[0007] To solve this problem, Patent Document 1 discloses a method for testing parallel vulnerabilities that detects read and write accesses of programs to shared memory.

[0008] First, we identify test cases for a program using a gray-box fuzzer. Then, based on the identified test cases, we analyze two or more branches of the program that contain sets of conflicting pairs to determine whether the test cases are preferred test cases.

[0009] Then, in response to the test case being a priority test case, the test case is provided from the gray-box fuzzer to a concurrency checking module, which uses the test case according to one or more scheduling policies to perform testing. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Publication No. 2019-204482 Summary of the Invention [Problem to be solved by the invention]

[0011] The method of Patent Document 1 detects defects caused by conflicting memory states by using a technique such as model checking on the generated test cases to cover the execution order between threads.

[0012] In this way, model checking can comprehensively check the execution order of multiple threads and verify whether the above-mentioned problems occur.

[0013] However, in large-scale systems, the number of execution orders between threads becomes enormous, and the number of conflicting memory states to be checked also becomes enormous, and model checking may not be completed within the limited time.

[0014] In order to detect defects caused by conflicting memory states within a limited time, it is desirable to give priority to checking conflicting memory states that are likely to occur.

[0015] Therefore, it is necessary to prioritize and inspect conflicting memory states based on the likelihood of occurrence.

[0016] In Patent Document 1, the execution order between threads is covered and a test is performed to check whether any malfunction occurs in all conflicting memory states, resulting in an enormous amount of test cases. [Means for solving the problem]

[0017] The above object can be achieved by a software testing device for software in which memory conflicts may occur, the processing device comprising a scenario processing unit, a test result processing unit, a conflict confirmation processing unit, and a state table generation processing unit, the scenario processing unit executes a test program of a test scenario including a test program for testing the software stored in a storage device and expected results obtained when the test program is executed, the test result processing unit determines whether the results of executing the test program match the expected results and records the test result information in the storage device, the conflict confirmation processing unit detects memory conflicts in which multiple programs constituting the software write different values ​​to the same address when the test program is executed, and the state table generation processing unit creates a state table in the storage device indicating test cases for each conflicting memory state to be tested based on the memory conflicts detected by the conflict confirmation processing unit. [Effects of the Invention]

[0018] It reduces the testing time for software that experiences memory conflicts. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating a system configuration of a test system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a test device according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating an example of a hardware configuration of a test device according to an embodiment of the present invention. [Figure 4] Examples of source code to be inspected in the embodiment of the present invention [Figure 5] Examples of test scenarios in embodiments of the present invention [Figure 6]Example of thread writing information in an embodiment of the present invention [Figure 7] Examples of memory contention ranges in an embodiment of the present invention [Figure 8] Example of a conflict memory status table in an embodiment of the present invention [Figure 9] Example of similarity calculation result of memory conflict pattern in the embodiment of the present invention [Figure 10] Example of weighted similarity calculation result of memory conflict patterns in an embodiment of the present invention [Figure 11] 10 is an example of a flowchart showing the processing of a test device in an embodiment of the present invention. [Figure 12] 10 shows an example of an output screen in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiments, the same components are given the same names and reference numerals as much as possible, and repeated explanations thereof will be omitted.

[0021] The present invention is not limited to the following examples, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above examples have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations.

[0022] Furthermore, the processing units and processing modules described in the embodiments may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0023] The information explained in the embodiment may be a table, a database (DB), or data stored in the main memory. [Example]

[0024] 1 is a diagram illustrating an example of a system configuration of a test system according to an embodiment of the present invention.

[0025] The test device 1 is connected to a test computer 3 via a network 2. The test computer may be the same computer as the test device, but in that case, it is necessary to provide an independent memory area to be used so that the operation of the test device does not affect the processing of the test computer.

[0026] The test computer is provided with an operation interface 4 that receives instructions from the test device for starting programs and setting memory states.

[0027] An operation instruction such as start or stop is given to the program 5 under test from the operation interface, and the thread 7 constituting the program under test runs.

[0028] Writing to the shared memory 6 is performed from the thread using an operating system interface or system call such as Write().

[0029] If this write were to the same address, a memory conflict would occur.

[0030] FIG. 2 is a block diagram showing an example of the configuration of a test device according to an embodiment of the present invention.

[0031] In the test device 1, the scenario processing unit 21 reads the test scenario 30 including the source code of the test program and the expected results obtained when the test program is executed, and writes the read scenario 30 to the thread writing information 31.

[0032] The written test scenario is executed by the scenario processing unit 21, and memory conflict detection is performed by the conflict confirmation processing unit 22. The result of the memory conflict detection is stored as conflict range information 32.

[0033] Based on the conflict range information 32, the status table generation processing unit 24 creates a conflict memory status table 34. Furthermore, the test result processing unit 23 compares the test results with the expected results, determines whether the test passed or failed, and stores the determined result in test result information 33.

[0034] Based on test result information 33 indicating whether the test result was successful or not and a conflict memory status table 34 indicating the memory conflict status, the next status selection processing unit 25 creates next status information 35 that stores the test program to be tested next and the memory status.

[0035] The state incorporation processing unit 26 registers the created next state information 35 in the test scenario 30, and the next test is executed.

[0036] These processing units and information do not necessarily have to be divided as shown in the block diagram in Figure 2; multiple processing units may be realized as a single processing unit, or each processing unit may be further divided into multiple processing units.

[0037] Furthermore, the information does not need to be structured as independent pieces of information; all information may be stored in one database, or the divided information may be associated and arranged separately.

[0038] FIG. 3 is a block diagram showing an example of the hardware configuration of a test device in an embodiment of the present invention.

[0039] The test device is realized by a computer having a CPU (Central Processing Unit) 40, memory 41 such as ROM (Read Only Memory) or RAM (Random Access Memory), input / output devices 43 such as a display, keyboard, mouse, etc., external storage devices 42 such as HDD (Hard Disk Drive) or SSD (Solid State Drive), and a network interface 44 such as NIC (Network Interface Card).

[0040] The memory 41 stores the scenario processing unit 21, test result processing unit 23, conflict confirmation processing unit 22, state table generation processing unit 24, next state selection processing unit 25, and state incorporation processing unit 26 as software modules, which are executed by the CPU 40 by referring to information such as tables stored in the external storage device 42.

[0041] The CPU 40 and the memory 41 are collectively called a processing unit, and the memory 41 and the external storage device are collectively called a storage device.

[0042] The external storage device 42 stores a test scenario 30 including a test program, expected results, memory status, etc., thread write information 31, conflict range information 32, test result information 33, and conflict memory status table 34.

[0043] The scenario processing unit 21 acquires a test scenario from the test scenario unit 30 and executes instructions written in the test scenario one by one.

[0044] The test result information 33 stores the test results obtained by executing the instructions and information about the state of the memory of the test device 1 after the test.

[0045] Among the instructions executed in the test scenario, the values ​​written to the shared memory 6 of the thread obtained by the monitoring instruction are stored in the thread write information 31 and are referred to as needed.

[0046] The conflict confirmation processing unit 22 acquires values ​​written from threads to the shared memory 6 from the thread write information 31 during execution of the test scenario, and identifies the addresses and values ​​of the memory written by multiple threads.

[0047] If multiple values ​​are written to the specified address, the conflict confirmation processing unit 22 stores information about the conflict range and the values ​​written to the memory in conflict range information 32, as a memory conflict that occurred in a test using a test scenario.

[0048] The test result processing unit 23 acquires from the scenario processing unit 21 the pass / fail result of the test, which indicates whether the result of executing the test scenario matches the expected result, and acquires the conflict range from the conflict range information 32. Based on the acquired conflict range, the test result processing unit 23 acquires the value of the memory to be tested, generates test results, and stores them in the test result information 33.

[0049] When a test scenario is executed, the state table generation processing unit 24 generates combinations of values ​​that may be written from sets of values ​​that may be written to the conflict range obtained from the conflict range information 32, and creates a combination table.

[0050] The created combination table is stored in the external storage device 42 as a conflict memory state table 34. In this example, the range of combinations is not limited, but in addition to the method of generating states comprehensively, the test time can be reduced by using a pairwise method that limits the range of combinations and generates test patterns efficiently.

[0051] The next-state selection processing unit 25 acquires the test results and the conflict memory state table 34 from the test result information 33, calculates the similarity between each pattern in the conflict memory state table and the memory state described in the test results, and selects one or more conflict memory states as the next state based on the similarity, which is then held in the next-state selection processing unit 25.

[0052] In this case, there are various methods for calculating the similarity, such as Hamming distance or counting the number of different states, but this embodiment is not limited to such calculation methods.

[0053] The state incorporation processing unit 26 receives the test scenario executed by the scenario processing unit 21 and the memory state to be executed next from the test scenario 30 and the next state selection processing unit 25, and generates a new test scenario in which the monitoring end instruction of the test scenario is changed to a memory rewrite instruction.

[0054] The newly generated test scenario is stored in the test scenario 30. The processing such as setting the value of the conflict memory performed by the state incorporation processing unit 26 may be performed by the scenario processing unit.

[0055] FIG. 4 shows an example of source code to be inspected in the embodiment of the present invention.

[0056] The source code 50 is written in C language and includes a process 54 for Thread1 and a process 55 for Thread2.

[0057] The processing of each thread may be executed from process 51 on line 5 to process 52 on line 6, or may branch from process 51 on line 5 to process 53 on line 10 depending on the value of a variable stored in memory 41 at the time of execution.

[0058] The scenario processing unit 21 acquires the test scenario 30, executes the process 54 of Thread 1 and the process 55 of Thread 2 based on the instructions written in the scenario, and monitors the memory 41.

[0059] FIG. 5 is an example of a test scenario in an embodiment of the present invention.

[0060] 10 is an example of a test scenario held by the test scenario 30 and provided to the scenario processing unit 21.

[0061] The test scenario 30 includes a test scenario item number 60, an instruction execution order 61, an instruction name 62, a waiting time until the instruction is given 63, parameters used for the instruction 64, and an executed 65 which is identification information indicating whether the instruction has been executed.

[0062] Note that the test scenario 30 includes test scenarios for verifying the operation of technologies and systems such as Selenium and Ranorex (registered trademark), and is not limited to those input to a specific application. In this embodiment, the following four instructions are shown as examples of instructions in a test scenario.

[0063] First, we will explain the target operation, which is the instruction shown as 2 in the execution order of the test scenario 30. This instruction is an instruction for the scenario processing unit 21 to perform an operation on the test target based on the content of the parameter 64, as is done in general software testing.

[0064] Next, the monitoring start instruction shown as 1 in the execution order of the test scenario 30 will be described.

[0065] This instruction is for the scenario processing unit 21 to start monitoring the memory 41 into which the program to be tested has been loaded.

[0066] This instruction allows monitoring of the state of memory written by process 54 of Thread 1 and process 55 of Thread 2 of the program under test, and this instruction is used to obtain the range and value written to memory 41 of each thread.

[0067] Next, the instruction for ending monitoring, which is indicated as instruction 4 in the execution sequence of the test scenario 30, will be described.

[0068] This instruction instructs the CPU 40 to wait for the progression of the scenario until all currently running threads have finished.

[0069] When all running threads are terminated by this instruction, the write range and value of each thread obtained by the monitoring start instruction are compiled into one piece of data and output as thread write information.

[0070] The thread write information output in response to this instruction is assigned the execution order of 4, which corresponds to the instruction to end monitoring of the test scenario 30, and is output.

[0071] Finally, we will explain the memory operation instruction. This instruction causes the CPU to wait until all running threads have finished, and then rewrites the value of the shared memory 6 based on the parameter 64.

[0072] It should be noted that this example of instructions is merely an example of an embodiment, and the content and type of instructions are not limited to this.

[0073] In this embodiment, at step 1 in the execution order, the program to be tested starts monitoring the shared memory 6 into which it has been loaded.

[0074] Next, in execution order 2, a target operation is performed to execute the process 54 of Thread 1 on the test target specified by the parameter 64.

[0075] Next, in execution order 3, after executing the instruction in execution order 2 above, the CPU 40 waits for 10 msec as specified in wait time 63 to execute the test scenario, and then performs a target operation to execute processing 55 of Thread 2 specified in parameter 64 on the test target.

[0076] Next, at execution order 4, the monitoring that started at execution order 1 is ended, and the range and values ​​of the writes made by each thread to shared memory 6 that occurred at execution order 2 and execution order 3 are output as thread write information.

[0077] Next, at execution order 5, after waiting for 10 msec described in waiting time 63 for execution of the test scenario by CPU 40, a target operation is performed to execute process 54 of Thread 1 on the test target specified by parameter 64.

[0078] FIG. 6 is an example of thread writing information in an embodiment of the present invention.

[0079] This is thread write information that is held by the thread write information 31 and provided to the conflict confirmation processing unit 22.

[0080] The thread write information 31 includes the execution order 61 of the monitoring end instruction in the test scenario, a thread ID 71 indicating which thread the information was written to, an address location 72 indicating the start and end addresses indicating the memory location where the writing took place, and write content 73 which is the content written to the address location 72.

[0081] FIG. 7 is an example of a memory contention range in an embodiment of the present invention.

[0082] 10 is an example of information held in conflict range information 32 used by the state table generation processing unit 24.

[0083] The conflict range information 32 includes an item number 80 assigned for identification, an identifier 81 of the conflict memory state in the execution order in which the conflict was monitored, and a conflict range 82 which is the address where the memory conflict occurred.

[0084] FIG. 8 is a diagram showing an example of a conflict memory status table in an embodiment of the present invention.

[0085] The conflict memory state table 34 stores combinations of execution order 90 instructed by the test scenario and values ​​stored in memory in conflict range 91, which is the address where memory conflict occurs.

[0086] FIG. 9 shows an example of the results of calculating the similarity of memory conflict patterns in the embodiment of the present invention.

[0087] The conflict memory state table 34 stores the similarities 92 and 93 of the memory values ​​in a specific conflict range 91 in association with each conflict memory state pattern that occurred in the test of execution order 4.

[0088] In this case, for similarity 1, 1 is compared with the memory value of the reference conflict memory state pattern, and the similarity is calculated as the value divided by the number of different bits.

[0089] In this example, the memory value in conflict memory state pattern 2 surrounded by the dotted line is used as the reference, and in conflict memory state pattern 1, only the value of address 0xA7 is different, so the similarity 1 is 1 / 1.

[0090] In this embodiment, similarity is used as the priority of testing in a conflicting memory state, and testing is performed based on the priority. However, more efficient testing can be performed by increasing the priority based on the frequency of occurrence of the conflicting memory state or the priority of a conflicting memory state that occurs when using a software function.

[0091] FIG. 10 shows an example of a result of calculating weighted similarity of memory conflict patterns in an embodiment of the present invention.

[0092] An example is given in which a priority (similarity) is calculated for each conflicting memory address with a weight 94. These weights can be used to prioritize tests based on the functionality provided by the software.

[0093] The priority of each software function may be determined not only based on whether the function is important, but also based on an index of how much the shared memory affects the quality of the software.

[0094] For each conflict memory state pattern that occurred in the test of execution order 4, the weight 94, the weighted memory state 95 calculated using the weight, and the similarity 2 calculated using the weighted state are stored in association with each other.

[0095] In this example, a weight of 3 is applied to address 0xA4 in the conflict range, a weight of 2 is applied to address 0xA6 in the conflict range, and a weight of 1 is applied to address 0xA7 in the conflict range. For conflict memory state pattern 1, the weighted memory state is 3*0+2*0+1*0=0.

[0096] When similarity 2 is calculated using the weighted memory state value, if conflict memory state pattern 2 surrounded by the dotted line is used as the reference, similarity 2 = 1 / (ΣWn|Xn-Xi|) (W is the importance, X is the value of the conflict range, n is the current pattern, i is the reference pattern), and is calculated as 1 / |0-1| = 1 / 1. FIG. 11 is an example of a flowchart showing the processing of the test device in the embodiment of the present invention.

[0097] The system accepts the selection of a scenario for starting software testing (S1), executes the selected scenario, and records the writing to memory during execution (S2).

[0098] The conflict confirmation processing unit 22 determines the addresses to which different values ​​are written from multiple threads as conflict ranges and registers them in the conflict range information 32 (S3).

[0099] Next, the test results of the scenario are compared with pre-registered results, and the pass / fail of the test is registered in the test result information (S4).

[0100] A conflict memory state table 34 is created to store a plurality of memory states created based on the similarity of memory values ​​from the conflict range information 32 (S5).

[0101] The test results of the received scenario are judged to be pass or not (S6), and if pass, the tests are sorted in descending order of similarity of memory state and the test with the most similar memory state is selected (S7).If fail, the tests are sorted in descending order of similarity of memory state and the test with the least similar memory state is selected (S8).

[0102] In this case, the similarity may take multiple values, and the calculation formula for the similarity may be other than the examples shown in FIGS.

[0103] The selected memory state is set in the memory of the test computer (S9), and the test is executed (S10). It is determined whether there is an untested conflict memory state in the conflict memory state table (S11), and if there is, the next memory state is set in the memory of the test computer (S12), the test is executed (S10), and the test result and the conflict memory value are stored in test result information 33.

[0104] If there is no untested conflict memory state, the test result and the conflict memory state are output in association with each other (S13).

[0105] It is not necessary to test all of the test cases in the conflict memory status table sorted in (S7) and (S8); by executing a predetermined number of test cases, it is possible to further reduce the test time.

[0106] Finally, it is determined whether there are any untested scenarios (S14), and if there are, the process returns to S2 to execute the untested scenarios.

[0107] FIG. 12 is an example of an output screen in an embodiment of the present invention.

[0108] The test progress and test results are output. In this example, the progress and test results for each conflict memory state pattern for execution order 4 of test scenario name S003 are displayed.

[0109] Test progress 101 indicating whether the test has already been completed, verification status 102 indicating whether verification has been performed, conflict memory state pattern 103, conflict range 104 indicating the value of the conflict memory, and test result 105 are displayed in association with each other.

[0110] The message field 106, which outputs a diagnostic message analyzing the test results, may output the analysis results of the test results and information on whether the execution order of the conflicting memory state patterns was determined based on the test result of the first executed test case being OK or NG.

[0111] If the first test case executed is NG, it is expected that the test result is NG due to the memory conflict, and it is also expected that the test result will be NG even if a test with a high degree of similarity in the conflicting memory is performed.

[0112] Therefore, an effective test strategy is to detect more problems by testing other test cases with low similarity in conflicting memories.

[0113] If the first test case is successful, it is expected that the software regarding the memory contention is written correctly.

[0114] Therefore, an effective testing strategy is to test other test cases that have a high degree of similarity to the memory conflict, and then to confirm that there are no software problems related to the memory conflict by executing test cases that have a high degree of similarity to the memory conflict. [Explanation of symbols]

[0115] 1 Test device, 2 Network, 3 Test computer, 4 Operation interface, 5 Test target program, 6 Shared memory, 7 Thread, 21 Scenario processing section, 22 Conflict check processing section, 23 Test result processing section, 25 Next state selection processing section, 26 State embedding processing section, 30 Test scenario, 31 Thread write information, 32 Conflict range information, 33 Test result information, 34 Conflict memory state table, 35 Next state information, 40 CPU, 41 Memory, 42 External storage device, 43 Input / output device, 44 Network interface

Claims

1. A test device for software in which memory contention may occur, The processing device comprises a scenario processing section, a test result processing section, a conflict checking processing section, and a state table generation processing section; the scenario processing unit executes a test program stored in the storage device, the test program being a test scenario including a test program for testing the software and an expected result to be obtained when the test program is executed; The test result processing unit determines whether the result of executing the test program matches the expected result, and records the test result information in the storage device. The conflict checking processing unit detects memory conflicts in which multiple programs constituting the software write different values ​​to the same address when the test program is executed, The state table generation processing unit is a software testing device that creates, in the storage device, a state table that indicates test cases for each conflicting memory state that is tested based on the memory conflict detected by the conflict confirmation processing unit.

2. 2. The software testing apparatus according to claim 1, The state table generation processing unit creates conflict memory state patterns, which are possible combinations of values ​​of the shared memory that cause the detected memory conflict, and determines the priority as a test case for each conflict memory state pattern. The scenario processing unit sets a conflict memory state pattern based on the priority and executes the test program.

3. 3. The software testing device according to claim 2, The software testing device has a priority that is a similarity between a conflict memory value and a conflict memory state pattern selected from a plurality of conflict memory state patterns included in the state table.

4. 4. The software testing device according to claim 3, The state table generation processing unit is a software testing device that calculates the similarity using a weight set for each address in the conflict memory when calculating the similarity.

5. 4. The software testing device according to claim 3, The state table generation processing unit is a software testing device that executes a test program from a test case of a highly similar conflict memory state pattern when the test result of the selected conflict memory state pattern matches the expected result.

6. 4. The software testing device according to claim 3, The state table generation processing unit is a software testing device that executes a test program from a test case of a conflict memory state pattern with a low similarity when the test result of the selected conflict memory state pattern does not match the expected result.

7. A method for testing software that may experience memory contention, comprising: The processing device comprises a scenario processing section, a test result processing section, a conflict checking processing section, and a state table generation processing section; The scenario processing unit executes a test program stored in the storage device, the test program being a test scenario including a test program for testing the software and an expected result obtained when the test program is executed; The test result processing unit determines whether the result of executing the test program matches the expected result, and records the test result information in the storage device; When the conflict checking processing unit executes the test program, it detects memory conflicts in which multiple programs constituting the software write different values ​​to the same address; A software testing method in which a state table generation processing unit creates in a storage device a state table indicating test cases for each conflicting memory state to be tested based on memory conflicts detected by a conflict confirmation processing unit.

Images