Software operation test device and software operation test method

Abstract

This software operation test device executes an operation test of subject software by observing the behavior of the subject software, and detects the test state corresponding to the state of the operation test being executed. The test state includes a first state indicating that coverage of the subject software in the operation test is increasing, a second state indicating that the increase in the coverage is stopped, a third state indicating a grace period until the end of the operation test, and a fourth state indicating the end of the operation test. The test state is detected such that the test state transitions from the first state to the fourth state via the second state and the third state.

Description

Software operation test device and software operation test method

[0001] The present invention relates to an apparatus and method for testing software operation.

[0002] A testing technique called fuzzing has been known for testing the operation of software during the development stage. Fuzzing is a testing technique that uses a testing tool called a fuzzer to automatically generate large amounts of data that may cause problems in the target software, such as invalid data, unexpected data, or random data, and input it into the target software. By observing the behavior of the target software at this time, it is possible to discover defects and vulnerabilities in the target software. In fuzzing, the testing data generated by the fuzzer is called fuzz data.

[0003] Generally, fuzzing is performed by sequentially inputting large amounts of randomly generated fuzz data into target software according to configuration parameters, and recording the information in a log when abnormal behavior occurs in the target software. In this type of fuzzing, the coverage of the target software (the number of executed code branches) increases with the amount of fuzz data input, but after a certain level of coverage plateaus, and further input of fuzz data no longer improves coverage. Continuing fuzzing in this state results in a waste of system resources required to run the target software. Therefore, it is preferable to interrupt fuzzing once coverage has stabilized and no further improvement is expected, and to change the fuzzer's configuration parameters, etc.

[0004] A known technique for automatically suspending fuzzing is described in Patent Document 1. Patent Document 1 describes a method for setting thresholds in advance for the duration of fuzzing and the number of test cases, and suspending fuzzing when these values ​​reach the thresholds.

[0005] U.S. Pat. No. 11,366,748

[0006] The method of Patent Document 1 stops fuzzing when the execution time or number of test cases reaches a threshold, regardless of the coverage status of the target software. As a result, there is a risk that fuzzing will stop early even if the coverage of the target software is insufficient, or that fuzzing will continue unnecessarily even if further improvement in coverage is not expected.

[0007] The present invention has been made based on this background, and aims to provide a technology that can stop fuzzing at an appropriate timing.

[0008] A software operation testing device according to the present invention comprises a target software acquisition unit that acquires target software to be the subject of an operation test; an operation test execution unit that executes an operation test of the target software by observing the behavior of the target software when fuzz data generated by a fuzzer is input to the target software; and a test state detection unit that detects a test state corresponding to the state of the operation test being executed by the operation test execution unit, wherein the test states include a first state indicating that the coverage of the target software in the operation test is increasing, a second state indicating that the increase in coverage has stopped, a third state indicating a grace period until the end of the operation test, and a fourth state indicating the end of the operation test, and the test state detection unit detects the test state so as to transition from the first state to the fourth state via the second state and the third state. A software operation testing method according to the present invention involves using a computer to acquire target software to be subjected to an operation test, observing the behavior of the target software when fuzz data generated by a fuzzer is input to the target software, thereby performing an operation test of the target software, and detecting a test state corresponding to the state of the operation test being executed, wherein the test states include a first state indicating that the coverage of the target software in the operation test is increasing, a second state indicating that the increase in coverage has stopped, a third state indicating a grace period until the end of the operation test, and a fourth state indicating the end of the operation test, and the test state is detected so as to transition from the first state to the fourth state via the second state and the third state.

[0009] According to the present invention, a technique can be provided that can stop fuzzing at an appropriate timing.

[0010] A diagram showing an example of the configuration of a software operation testing device according to an embodiment of the present invention. A block diagram showing the functional configuration of a software operation testing device according to an embodiment of the present invention. A flowchart showing the processing flow of the software operation testing device. An explanatory diagram of test state detection processing. A flowchart showing the flow of test state detection processing.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] FIG. 1 is a diagram showing an example of the configuration of a software operation test device according to an embodiment of the present invention.

[0013] 1 is a software operation testing device 1 that performs an operation test of target software using fuzzing. The software operation testing device 1 is configured using an information processing device (computer) such as a server or a PC, and includes a processor 10, a memory 20, a recording device 30, a network interface 40, an input device 50, and an output device 60. In the software operation testing device 1, these devices are connected to each other via a communication bus.

[0014] The processor 10 executes a predetermined program to control the operation of the software operation test apparatus 1 and to cause a computer such as a server or PC to function as the software operation test apparatus 1. The processor 10 is configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc. The memory 20 is a volatile storage medium used as a work area when the processor 10 executes a program. The recording device 30 is a recording medium capable of storing various programs and data and is configured using, for example, a hard disk drive (HDD), a solid state drive (SSD), etc. The network interface 40 is a part that performs interface processing for data input / output between the software operation test apparatus 1 and a network (not shown) (e.g., a LAN, the Internet, etc.). Software to be tested is input to the software operation test apparatus 1 via the network interface 40, recorded in the recording device 30, and used in processing performed by the processor 10. The input device 50 is a device that accepts data input to the software operation test device 1 from a user who performs an operation test of the target software using the software operation test device 1, and is configured using, for example, a keyboard, a mouse, a touch panel, a microphone, etc. The output device 60 is a device that outputs information to the user, and is configured using, for example, a display, a printer, a speaker, etc.

[0015] 2 is a block diagram showing the functional configuration of a software operation testing apparatus 1 according to an embodiment of the present invention. As shown in FIG. 2, the software operation testing apparatus 1 includes the following functional blocks: a target software acquisition unit 11, a preprocessing execution unit 12, a fuzzer 13, an operation test execution unit 14, a log information recording unit 15, a frontier detection unit 16, a test status detection unit 17, and an information presentation unit 18. These functional blocks are realized by the processor 10 in FIG. 1 reading out a program stored in the recording device 30 and executing processing in accordance with this program.

[0016] The software operation test device 1 also stores the following information: target software 31, software configuration information 32, log information 33, frontier information 34, and test status detection information 35. These pieces of information are stored in the recording device 30 of FIG. 1 in the software operation test device 1.

[0017] The target software acquisition unit 11 acquires information about the target software for the operation test via the network interface 40. For example, the information about the target software is acquired by a binary file expressed in machine code. The information about the target software acquired by the target software acquisition unit 11 is stored in the recording device 30 as target software 31.

[0018] The preprocessing execution unit 12 analyzes the configuration of the target software 31 (combinations of basic blocks of instructions and code branches branching from each basic block) by executing predetermined preprocessing on the target software 31. The configuration of the target software 31 analyzed by the preprocessing execution unit 12 is stored in the recording device 30 as software configuration information 32.

[0019] The fuzzer 13 generates fuzz data to be input to the target software 31. While executing an operation test of the target software 31, the fuzzer 13 generates fuzz data based on, for example, the software configuration information 32, so as to reduce the number of unexecuted code branches in the target software 31 as much as possible and improve coverage. Note that the fuzz data generated by the fuzzer 13 can use, for example, a well-known technique called a gray box.

[0020] The operation test execution unit 14 inputs the fuzz data generated by the fuzzer 13 into the target software 31, executes the target software 31, and observes the behavior of the target software 31 at this time, thereby executing an operation test of the target software 31. The observation results of the behavior of the target software 31 by the operation test execution unit 14 are stored in the recording device 30 as log information 33 by the log information recording unit 15.

[0021] The frontier detection unit 16 detects frontiers in the operational test of the target software 31 while the operational test execution unit 14 is executing the operational test of the target software 31. Specifically, among the code branches branching from each basic block of the target software 31, the frontier detection unit 16 detects a code branch in which the instructions of the basic block at the branch source of the code branch have been executed in the operational test but the code branch has not yet been executed as a frontier in the operational test of the target software 31. Information on each frontier detected by the frontier detection unit 16 is stored in the recording device 30 as frontier information 34.

[0022] The test status detection unit 17 detects the test status that indicates the status of the operational test being executed by the operational test execution unit 14. As a result, the current state of the operational test being executed is detected as the test status. Details of how the test status detection unit 17 detects the test status will be described later. The test status detection result by the test status detection unit 17 is stored in the recording device 30 as test status detection information 35.

[0023] The information presentation unit 18 presents information such as the status of an operational test currently being performed and the results of an operational test that has already been performed to the user of the software operation test device 1 using the output device 60. For example, the information can be provided to the user by displaying characters or images indicating the content on a display included in the output device 60.

[0024] 3 is a flowchart showing the flow of processing performed by the software operation test device 1. The processing shown in this flowchart is performed by the processor 10, for example, at predetermined processing intervals.

[0025] In step S10 , the target software acquisition unit 11 acquires software to be subjected to the operation test, and stores it in the recording device 30 as target software 31 .

[0026] In step S20, the preprocessing execution unit 12 executes preprocessing on the target software 31 stored in the recording device 30 in step S10, and generates software configuration information 32.

[0027] In step S30, fuzzing (operational testing) is performed on the target software 31 using the fuzzer 13, the operation test execution unit 14, and the log information recording unit 15. Specifically, while the target software 31 is executed by the operation test execution unit 14, fuzz data generated by the fuzzer 13 is input to the target software 31, and log information 33 related to the behavior of the target software 31 at this time is stored in the recording device 30 by the log information recording unit 15, thereby performing an operational test of the target software 31. In this way, fuzzing of the target software 31 is performed.

[0028] In step S40, while fuzzing (operational testing) is being performed on the target software 31 in step S30, the frontier detection unit 16 detects frontiers in the operational test of the target software 31. As described above, among the code branches branching from each basic block of the target software 31, those in which the instructions of the basic block at the branch source of the code branch have been executed in the operational test but the code branch has not yet been executed are detected as frontiers in the operational test of the target software 31. Then, information on each detected frontier is stored in the recording device 30 as frontier information 34.

[0029] In step S50, the test status detection unit 17 performs a test status detection process for the operation test of the target software 31 being performed in step S30. In this test status detection process, it is determined whether the test status of the operation test being performed corresponds to "start," "rising," "saturation," "grace period," "warning," or "end," and the determination result is stored in the recording device 30 as test status detection information 35.

[0030] "Start" indicates that the operational test has just started, and quickly transitions to "Increasing" as the operational test progresses. "Increasing" indicates that the coverage of the target software 31 in the operational test, i.e., the number of executed code branches, is increasing. "Saturated" indicates that the increase in coverage of the target software 31 in the operational test has stopped. "Grace" indicates a state in which a grace period is given before the operational test ends, after the increase in coverage of the target software 31 in the operational test has stopped. "Alert" indicates a state in which the information presenting unit 18 outputs an alert to the user, prompting the user to decide whether to end the operational test and waiting for instructions from the user. "End" indicates a state in which the operational test should be ended, and when the test status detecting unit 17 determines that the operational test is in this state, the execution of the target software 31 is stopped and the operational test is quickly ended.

[0031] The test state detection process in step S50 will be described in detail later with reference to FIGS.

[0032] In step S60, it is determined whether the test state obtained by the test state detection process in step S50 is “alarm.” If the test state is “alarm,” the process proceeds to step S70, and if the test state is any other than “alarm,” the process proceeds to step S80.

[0033] In step S70, a warning is output to the user. Here, the information presenting unit 18 outputs, for example, a predetermined image or sound as the warning to the user. By this warning, the software operation testing device 1 prompts the user to confirm that the coverage increase in the operation test of the target software 31 has stopped, and prompts the user to decide whether to end the operation test or continue it.

[0034] The user, having confirmed the warning output in step S70, decides whether or not to end the operation test of the target software 31, and if he or she decides to end it, performs a predetermined intervention operation using the input device 50. When this intervention operation is performed, the software operation test device 1 determines that the test status of the operation test being performed is "ended" by the test status detection process in step S50, as will be described later.

[0035] In step S80, it is determined whether the test status obtained by the test status detection process in step S50 is "Ended." If the test status is "Ended," the process shown in the flowchart in Fig. 3 is terminated. If the test status is other than "Ended," the process returns to step S30 and continues the process shown in the flowchart in Fig. 3. As a result, if the test status is other than "Ended," fuzzing (operation testing) of the target software 31 continues.

[0036] Next, the test state detection process in step S50 will be described in detail below.

[0037] 4 is an explanatory diagram of the test state detection process, in which (a) shows state transitions between test states in the test state detection process, and (b) shows transition conditions for the state transitions in (a).

[0038] In the test status detection process, when an operation test of the target software 31 is started, the test status is first determined to be "started." Then, as the coverage increases, the transition condition for state transition Lr is satisfied, and the test status changes from "started" to "increasing," as shown in FIG. 4A. While the coverage is increasing, the test status continues to be determined to be "increasing."

[0039] If the coverage stops increasing while the test state is "increasing," the transition condition for state transition Ls is determined to be met, and the test state changes from "increasing" to "saturated." If the coverage then increases again, the transition condition for state transition Lr is met, and the test state returns from "saturated" to "increasing."

[0040] On the other hand, when the test state is "saturated," if the elapsed time st from the start of the operational test exceeds the specified time spt of all frontiers detected by the frontier detection unit 16 at that time, it is determined that the transition condition for the state transition Lg has been met, and the test state transitions from "saturated" to "grace." Also, when the test state is "saturated," if the elapsed time st from the start of the operational test exceeds the predetermined total allocated time tat, it is determined that the transition condition for the state transition Lt has been met, and the test state transitions from "saturated" to "end."

[0041] Here, the above-mentioned specified time spt is determined for each frontier based on the execution difficulty of each code branch of the target software 31 in the operation test. In other words, the specified time spt for each frontier is calculated based on the execution difficulty previously set for each code branch detected as a frontier. Specifically, the specified time spt(f) for an arbitrary frontier f can be calculated by the following formula (1), where b is the code branch corresponding to the frontier f and cd(b) is the execution difficulty of this code branch b. spt(f) = cd(b) * tat / Σ(cd) ... (1)

[0042] In formula (1), Σ(cd) represents the total value of the execution difficulty of all code branches in the target software 31. Furthermore, tat represents the total allocated time for the operational test of the target software 31, which can be set to any value by the user in advance.

[0043] The execution difficulty cd(b) of code branch b can be set according to the depth from the code branch that is executed first in the target software 31 to code branch b, the length of code branch b, the type of operand corresponding to the execution condition of code branch b, etc. Alternatively, a default value (for example, 1) may be set.

[0044] For example, in the preprocessing executed by the preprocessing execution unit 12 in step S20 of Fig. 3, the specified time spt is calculated for each code branch of the target software 31 using equation (1). Then, software configuration information 32 including the calculation results of the specified time spt for each code branch is stored in the recording device 30. In this way, the test state detection unit 17 can obtain the specified time spt for each frontier from the software configuration information 32 in the test state detection process and use it to determine the transition condition of the state transition Lg.

[0045] If the coverage increases again while the test state is in "Grace," the transition condition for state transition Lr is satisfied, and the test state returns from "Grace" to "Increasing." Meanwhile, if the alarm determination value va, which is determined according to the elapsed time st since the start of the operational test at that time, falls below a predetermined threshold while the test state is in "Grace," it is determined that the transition condition for state transition La has been satisfied, and the test state transitions from "Grace" to "Alarm." Furthermore, if the elapsed time st since the start of the operational test exceeds a predetermined total allocated time tat while the test state is in "Grace," it is determined that the transition condition for state transition Lt has been satisfied, and the test state transitions from "Grace" to "End."

[0046] Here, the warning determination value va is determined based on the execution difficulty cd previously set for each code branch not yet executed in the operation test and the total allocated time tat allocated to the operation test of the target software 31. Specifically, the warning determination value va can be calculated, for example, by the following equation (2): va = spt0 / (tat - st) (2)

[0047] In formula (2), spt0 represents the largest specified time spt of each code branch that has not yet been executed in the operational test. The specified time spt of each code branch can be calculated based on the execution difficulty cd of each code branch using formula (1) above. As mentioned above, st represents the elapsed time since the start of the operational test.

[0048] If the coverage increases again while the test state is "alarm," the transition condition for state transition Lr is satisfied, and the test state returns from "alarm" to "increasing." On the other hand, if the test state is "alarm" and a user intervention operation is input, the number of alarm outputs exceeds a predetermined value, or the elapsed time st from the start of the operational test exceeds a predetermined total allocated time tat, it is determined that the transition condition for state transition Lat is satisfied, and the test state changes from "alarm" to "end."

[0049] In the test state detection process, as described above, the test state transitions from "start" through "rising," "saturation," and "grace period," before finally transitioning to "end." However, if the transition condition for state transition Lt is met during "saturation," the state transitions directly to "end" without passing through "grace period." Furthermore, if the transition condition for state transition La is met during "grace period," the state transitions to "end" after passing through "alarm."

[0050] FIG. 5 is a flowchart showing the flow of the test state detection process.

[0051] In step S510, the previous test status is referenced. Here, the test status detection information 35 is read to reference the test status detected in the previous step S50 in the processing flow of Fig. 3, and it is confirmed whether the previous test status was "start," "rising," "saturation," "grace period," "warning," or "end."

[0052] In step S520, the transition conditions for each possible test state from the previous test state referenced in step S510 are identified. Here, the possible test states to which the previous test state can be transitioned are confirmed based on the state transition diagram in FIG. 4A. The transition conditions for each possible test state are then identified based on the transition conditions in FIG. 4B. For example, if the previous test state was "rising," the transition condition shown in FIG. 4B is identified as the transition condition for the state transition Ls from the previous test state to "saturated." Similarly, for example, if the previous test state was "saturated," the transition conditions shown in FIG. 4B are identified as the transition conditions for the state transitions Lr, Lg, and Lt from the previous test state to "rising," "grace," and "end." Furthermore, for example, if the previous test state was "grace," the transition conditions shown in FIG. 4B are identified as the transition conditions for the state transitions Lr, Lg, and Lt from the previous test state to "rising," "alert," and "end." In other cases as well, the transition conditions to each possible test state from the previous test state can be identified using the state transition diagram in FIG. 4(a) and the transition conditions in FIG. 4(b).

[0053] In step S530, it is determined whether any of the transition conditions to each test state identified in step S520 is satisfied. If any of the transition conditions is satisfied, the process proceeds to step S540, where the test state determination result is changed to the corresponding test state, and then the process proceeds to step S550. On the other hand, if none of the transition conditions is satisfied, the process proceeds to step S550 without executing step S540.

[0054] In step S550, the process waits for a predetermined time (for example, one second). During this wait, it is preferable to continue the process of step S30 in FIG. 3 and continue fuzzing (operation testing) the target software 31.

[0055] In step S560, the specified time spt for each code branch is updated. Here, for example, the specified time spt for each code branch can be updated using the specified time of another code branch with the same execution difficulty cd as that of the code branch. Note that the processing of step S560 may be performed as needed and may be omitted. Furthermore, whether or not to perform the processing of step S560 may be switched for each code branch. For example, it is preferable to exclude a code branch for which there is no other code branch with the same execution difficulty cd from the processing target of step S560.

[0056] Once the process of step S560 has been performed or omitted, the test state detection process shown in the flowchart of FIG. 5 ends.

[0057] According to the embodiment of the present invention described above, the following advantageous effects can be obtained.

[0058] (1) The software operation testing device 1 includes a target software acquisition unit 11 that acquires target software 31 to be subjected to an operation test, an operation test execution unit 14 that executes an operation test of the target software 31 by observing the behavior of the target software 31 when fuzz data generated by a fuzzer 13 is input to the target software 31, and a test state detection unit 17 that detects a test state corresponding to the state of the operation test being executed by the operation test execution unit 14. The test states detected by the test state detection unit 17 include a first state (“increasing”) indicating that the coverage of the target software 31 in the operation test is increasing, a second state (“saturated”) indicating that the increase in coverage has stopped, a third state (“grace”) indicating a grace period until the end of the operation test, and a fourth state (“end”) indicating the end of the operation test. The test state detection unit 17 detects the test state so that the test state transitions from the first state to the fourth state via the second state and the third state. This prevents fuzzing from being stopped early despite insufficient coverage of the target software, or from continuing unnecessarily despite the fact that further improvement in coverage is not expected, thereby providing a technology that can stop fuzzing at the appropriate time.

[0059] (2) If the previous test state detection result is the second state, the test state detection unit 17 transitions the test state detection result from the second state to the third state when the transition condition of the state transition Lg is satisfied, that is, when the elapsed time st from the start of the operational test exceeds the specified time spt of all frontiers in the operational test of the target software 31. This allows the transition from the second state to the third state to be performed at an appropriate timing.

[0060] (3) The specified time spt is determined for each frontier using the above-mentioned formula (1) based on the execution difficulty cd of each code branch of the target software 31 in the operation test. In this way, the value of the specified time spt can be appropriately determined according to the execution difficulty cd of each frontier of the target software 31 during the operation test, i.e., each code branch for which the instructions of the basic block at the branch source of the code branch have been executed in the operation test but the code branch has not yet been executed.

[0061] (4) The software operation test device 1 includes an information presentation unit 18 that outputs an alert to the user. The test state detected by the test state detection unit 17 includes a fifth state ("alert") that causes the information presentation unit 18 to output an alert. If the previous test state detection result is the third state, the test state detection unit 17 transitions the test state detection result from the third state to the fifth state when the transition condition of the state transition La is satisfied, i.e., when the elapsed time st since the start of the operation test satisfies a predetermined alert condition. Specifically, the test state detection result transitions from the third state to the fifth state when a alert condition set based on the execution difficulty cd of each code branch not executed in the operation test and the total allocated time tat allocated to the operation test is satisfied, i.e., when the alert determination value va calculated by the above-mentioned equation (2) falls below a predetermined threshold. This configuration allows an alert to be output to the user at an appropriate time, allowing the user to decide whether to end the operation test, and waiting for instructions from the user.

[0062] (5) If the previous test state detection result is the second state or the third state, the test state detection unit 17 transitions the test state detection result from the second state to the fourth state without passing through the third state, or from the third state to the fourth state, when the transition condition for the state transition Lt is satisfied, i.e., when the elapsed time st since the start of the operational test has exceeded the total allocated time tat allocated to the operational test. In this way, the operational test of the target software 31 can be immediately terminated when the total allocated time tat allocated to the operational test of the target software 31, as set in advance by the user, has passed.

[0063] It goes without saying that the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0064] Furthermore, the above-described configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, an SD card, or a DVD.

[0065] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it can be considered that almost all components are actually connected to each other.

[0066] 1: Software operation test device 10: Processor 11: Target software acquisition unit 12: Preprocessing execution unit 13: Father 14: Operation test execution unit 15: Log information recording unit 16: Frontier detection unit 17: Test status detection unit 18: Information presentation unit 20: Memory 30: Recording device 31: Target software 32: Software configuration information 33: Log information 34: Frontier information 35: Test status detection information 40: Network interface 50: Input device 60: Output device

Claims

1. A software operation testing device comprising: a target software acquisition unit that acquires target software to be the subject of an operation test; an operation test execution unit that executes an operation test of the target software by observing the behavior of the target software when fuzz data generated by a fuzzer is input to the target software; and a test state detection unit that detects a test state corresponding to the state of the operation test being executed by the operation test execution unit, wherein the test states include a first state indicating that the coverage of the target software in the operation test is increasing, a second state indicating that the increase in coverage has stopped, a third state indicating a grace period until the end of the operation test, and a fourth state indicating the end of the operation test, and the test state detection unit detects the test state so as to transition from the first state to the fourth state via the second state and the third state.

2. A software operation testing device as described in claim 1, wherein, if the previous test state detection result is the second state, the test state detection unit transitions the test state detection result from the second state to the third state when the elapsed time since the start of the operation test exceeds the specified time for all frontiers in the operation test of the target software.

3. A software operation testing device according to claim 2, wherein the specified time is determined for each frontier based on the difficulty of executing each code branch of the target software in the operation test.

4. A software operation testing device as described in claim 1, comprising an information presentation unit that outputs an alert to a user, wherein the test state includes a fifth state that causes the information presentation unit to output the alert, and wherein if the previous detection result of the test state was the third state, the test state detection unit transitions the detection result of the test state from the third state to the fifth state when the elapsed time since the start of the operation test satisfies a predetermined alert condition.

5. A software operation testing device according to claim 4, wherein the warning conditions are set based on the execution difficulty of each code branch that has not been executed in the operation test and the total allocated time for the operation test.

6. A software operation testing device as described in claim 1, wherein, when the previous test state detection result is the second state or the third state, the test state detection unit transitions the test state detection result from the second state to the fourth state without passing through the third state, or from the third state to the fourth state, when the elapsed time since the start of the operation test has exceeded the total allocated time allocated to the operation test.

7. A software operation testing method in which a computer acquires target software to be the subject of an operation test, performs an operation test of the target software by observing the behavior of the target software when fuzz data generated by a fuzzer is input to the target software, and detects a test state corresponding to the state of the operation test being executed, wherein the test states include a first state indicating that the coverage of the target software in the operation test is increasing, a second state indicating that the increase in coverage has stopped, a third state indicating that there is time until the operation test is completed, and a fourth state indicating the completion of the operation test, and the test state is detected so as to transition from the first state to the fourth state via the second state and the third state.

Images