Scenario test device, scenario test method, and scenario test program

Abstract

A scenario test device according to one embodiment executes a scenario test on an object system, and comprises a mimicking unit and a searching unit. The mimicking unit uses multiple scenario test histories as training data to learn selections of an operation event by a user who performs an operation of the object system. The searching unit performs a search test, which is a trial for searching for defects due to execution of an operation and a screen transition, by: giving higher priority to operations with a lower probability among the learned selections; or giving a higher assessment to an operation in an operation scenario in which a defect is discovered.

Description

[Technical field]

[0001] An embodiment of the present invention relates to a scenario testing device, a scenario testing method, and a scenario testing program. [Background technology]

[0002] To discover system defects, scenario tests are performed on the system. Load testing tools are used for these scenario tests, and currently, various load testing tools are available, including Apache JMeter. With these existing load testing tools, human test engineers determine operation scenarios by assuming tasks that users will perform.

[0003] Monte Carlo tree search (hereinafter referred to as MCTS) is used as a method for predicting user operations. MCTS is a tree search method using the Monte Carlo method, and is a heuristic search algorithm for the decision process (it stops unnecessary searches midway and can lead to a good move with a relatively high probability). A major example of the use of MCTS is known to be the decision-making process for games such as computer Go, chess, and shogi (for example, see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] David Silver, et al., "Mastering the game of Go with deep neural networks and tree search", Nature, Vol. 529, 2016, pp. 484-489 Summary of the Invention [Problem to be solved by the invention]

[0005] To conduct scenario testing of a system using a load testing tool, a test engineer creates an operation scenario. In this case, the test engineer must decide the content of the scenario test while being aware of the process flow expected from human experience, such as past experience and normal work. This makes it difficult to find system malfunctions caused by unexpected user operations. In other words, it is not possible to respond to failures caused by unexpected operation scenarios.

[0006] It is also possible to anticipate user operations using MCTS, which is used in computer games such as Go, but because MCTS imitates actions that are likely to be chosen by humans, it is not good at dealing with unexpected operational scenarios.

[0007] In order to deal with failures due to such unexpected operational scenarios, it is conceivable to conduct comprehensive scenario testing. However, it would take a long time to conduct all possible scenario tests, which is not realistic.

[0008] The present invention provides a scenario testing technique that can efficiently discover system malfunctions caused by unexpected user operations. [Means for solving the problem]

[0009] In order to solve the above problems, a scenario testing device according to one aspect of the present invention is a scenario testing device that performs a scenario test on a target system, and includes an imitation unit and a search unit. The imitation unit learns selections for operation events of a user who operates the target system using multiple scenario test histories as training data. The search unit assigns a higher priority to operations with a lower probability among the learned selections, or increases the evaluation of an operation in an operation scenario in which a defect was found, and performs a search test, which is a trial to search for a defect by performing operations and screen transitions. Effect of the Invention

[0010] According to one aspect of the present invention, it is possible to confirm defects that exist in operations that a human is unlikely to perform on the target system, and it is also possible to test the scenario that is most likely to contain a defect for the entire system, thereby providing a scenario testing technology that can efficiently discover system defects caused by unexpected user operations. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a scenario testing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing an overview of MCTS. [Diagram 3] FIG. 3 is a block diagram showing an example of the configuration of a scenario testing system to which a scenario testing device according to an embodiment is applied. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of an engineer terminal to which a scenario testing device according to an embodiment is applied. [Diagram 5] FIG. 5 is a flowchart showing an example of a processing operation of the scenario testing device. [Figure 6] FIG. 6 is a schematic diagram showing an example of transition relationships between screens provided by the target system. [Figure 7] FIG. 7 is a schematic diagram showing an example of the screen in the example of FIG. 6 and the numbers given to operation events on the screen. [Figure 8] FIG. 8 is a schematic diagram for explaining an example of the operation of the imitator in FIG. [Figure 9] FIG. 9 is a schematic diagram for explaining an example of the operation of the searching unit and the determining unit in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] An embodiment of the present invention will now be described with reference to the drawings.

[0013] (Configuration example) 1 is a block diagram showing an example of the configuration of a scenario testing device 1 according to an embodiment of the present invention. The scenario testing device 1 performs a scenario test on a target system SYS. The target system SYS may be a hardware system, a software system, or a mixed system of hardware and software.

[0014] The scenario testing device 1 includes an initial value setting unit 2, a simulation unit 3, a search unit 4, and a judgment unit 5.

[0015] The test engineer TENG creates an operation scenario according to the target system SYS, and sets initial values ​​of various parameters based on the operation scenario by the initial value setting unit 2. The initial value setting unit 2 includes, for example, a screen number setting unit 21, an event number setting unit 22, a judgment criterion setting unit 23, a search test setting unit 24, and a scenario test setting unit 25. Note that in this embodiment, it is a prerequisite that the test engineer TENG knows all the screens and operation events in the target system SYS.

[0016] The screen number setting unit 21 receives from the test engineer TENG an order for the transition relationships between the screens of the target system SYS, and when the screen numbers assigned to each screen based on the order are input, the screen number setting unit 21 stores the screen numbers of each screen.

[0017] The event number setting unit 22 assigns a sequence to the operation events on each screen of the target system SYS from the test engineer TENG, and when the event numbers assigned to each operation event based on the sequence are input, the event number setting unit 22 stores the event numbers of each screen operation event.

[0018] The judgment criteria setting unit 23 receives a judgment criteria for a defect from the test engineer TENG and stores the judgment criteria for the defect.

[0019] The search test setting unit 24 receives the end conditions and rewards of the search test from the test engineer TENG, and stores the end conditions and rewards of the search test. In this embodiment, the attempt to search for defects by performing operations on the target system SYS and screen transitions is called a search test. In this embodiment, this search test is performed using MCTS.

[0020] FIG. 2 is a schematic diagram showing an overview of MCTS. MCTS consists of four steps: selection, expansion, evaluation, and backpropagation. By repeating this process multiple times (N times), the entire tree is searched for a promising location. The selection step is a step of selecting one leaf node of the search tree from the root node. The expansion step is a step of expanding child nodes from the selected leaf node and selecting one child node. The evaluation step is a step of performing random playout, starting from the selected child node C and repeating random expansion of child nodes until a termination condition is met. The update step is a step of propagating the result from the leaf node where the evaluation was performed toward the root R.

[0021] When performing a search test using such an MCTS, the end condition during the search can be artificially set by the depth of the MCTS, which is expected to make the search more efficient. The reward is the value returned as a result of searching up to the set MCTS depth. For example, the evaluation value of all actions and states of the search route where a fault occurred is given a "+1" and the evaluation list is updated. If there is no fault, all are given a "-1".

[0022] The scenario test setting unit 25 receives an end condition of the scenario test from the test engineer TENG, and stores the end condition of the scenario test.

[0023] The simulation unit 3 imitates operations of people who have experience in multiple scenario tests, which are indicated by multiple scenario test histories including test histories of multiple test engineers TENG and / or operation scenario histories such as standard scenarios created by the developer of the target system SYS, input from the target system SYS. In this simulation, the simulation unit 3 can use initial values ​​stored in the initial value setting unit 2. The simulation unit 3 includes, for example, a history learning unit 31 and a reversal unit 32.

[0024] The history learning unit 31 trains the policy network by history learning using behavior cloning, using multiple scenario test histories including past test histories and / or operation scenario histories as training data, i.e., by learning the selections made for operation events by users who operate the target system SYS.

[0025] The inverse unit 32 inverses the trained policy network that is the result of the learning.

[0026] The search unit 4 uses the inverted policy network obtained from the mimicry unit 3 to search for parts of the target system SYS that are least likely to be operated by a human and parts that are most likely to have a defect. In other words, by using the inverted policy network, the search unit 4 can perform a search in which the priority is increased for operations that are less likely to be selected by a user. In addition, by using the MCTS, the search unit 4 can perform a search by increasing the evaluation of operations in an operation scenario in which a defect is found. Specifically, the search unit 4 includes an MCTS search unit 41, an update unit 42, and a recording unit 43.

[0027] The MCTS search unit 41 uses the MCTS to perform a search test on the target system SYS, searching for parts that are least likely to be operated by a human and parts that are most likely to have defects. The MCTS search unit 41 outputs search information indicating the search test results to the judgment unit 5. In addition, upon receiving the end judgment result of the search test or scenario test from the judgment unit 5, the MCTS search unit 41 switches the target of the search test and the operation scenario for the target system SYS.

[0028] The update unit 42 receives the defect and reward judgment results from the judgment unit 5, and updates the evaluation list in accordance with the judgment results.

[0029] The recording unit 43 records the search test results by the MCTS search unit 41. The search test results can indicate the results of a defect determination.

[0030] The judgment unit 5 uses the initial values ​​stored in the initial value setting unit 2 to judge malfunctions, to judge the end of the search test and the reward, and to judge the end of the scenario test. The judgment unit 5 includes, for example, a result judgment unit 51, a search end judgment unit 52, and a scenario end judgment unit 53.

[0031] The result judgment unit 51 judges the defect and judges the reward for the search test. The result judgment unit 51 outputs the judgment result to the search unit 4. The result judgment unit 51 can also present the defect judgment result, that is, the part where the defect was found in the target system SYS, to the test engineer TENG. This allows the test engineer TENG to carry out a correction for the defect in the target system SYS based on the judgment result.

[0032] The search end judgment unit 52 judges whether the search test end condition is met and inputs the judgment result to the search unit 4. As a result, the MCTS search unit 41 repeatedly performs the search test until the search test end condition is met. When the search test end condition is met, the MCTS search unit 41 transitions to the next state in the scenario test based on the search test result recorded in the recording unit 43, and performs the search test.

[0033] The scenario end judgment unit 53 judges whether the end condition of the scenario test is met and inputs the judgment result to the search unit 4. As a result, the MCTS search unit 41 repeatedly performs the search test until the end condition of the scenario test is met.

[0034] 3 is a block diagram showing an example of the configuration of a scenario test system to which the scenario test device 1 is applied. The scenario system includes a target system SYS and an engineer terminal TER.

[0035] The engineer terminal TER is a computer to which the scenario test device 1 according to one embodiment is applied. The engineer terminal TER is operated by a test engineer TENG who determines an operation scenario for a scenario test on the target system SYS and corrects the target system SYS if a defect in the target system SYS is found in the scenario test.

[0036] The target system SYS and the engineer terminal TER can be connected directly or indirectly via a network. Also, the software part of the target system SYS may be implemented and executed on the engineer terminal TER.

[0037] FIG. 4 is a diagram illustrating an example of a hardware configuration of an engineer terminal TER to which a scenario testing device according to an embodiment is applied. The engineer terminal TER is configured by a computer such as a server computer or a personal computer, and has a hardware processor 100 such as a CPU (Central Processing Unit) as shown in FIG. 4. The CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded one. The processor 100 may also be equipped with multiple CPUs. In the engineer terminal TER, a program memory 200, a data memory 300, a communication interface 400, and an input / output interface (referred to as input / output IF in FIG. 3) 500 are connected to the processor 100 via a bus 600.

[0038] The communication interface 400 may include, for example, one or more wired or wireless communication modules. The communication interface 400 can communicate with the target system SYS connected via a cable or a network such as a LAN (Local Area Network) or the Internet, or with the target system SYS configured in an external data processing device such as another computer or server.

[0039] The input unit 700 and the display unit 800 are connected to the input / output interface 500. The input unit 700 and the display unit 800 may be a so-called tablet-type input / display device in which an input detection sheet using an electrostatic method or a pressure method is arranged on a display screen of a display device using, for example, liquid crystal or organic EL (Electro Luminescence). The input unit 700 and the display unit 800 may be configured as independent devices. The input / output interface 500 inputs operation information inputted in the input unit 700 to the processor 100, and causes the display unit 800 to display display information generated by the processor 100.

[0040] The input unit 700 and the display unit 800 do not have to be connected to the input / output interface 500. The input unit 700 and the display unit 800 can transmit and receive information to and from the processor 100 by including a communication unit for connecting to the communication interface 400 directly or via a network.

[0041] The input / output interface 500 may also have a function for reading / writing a recording medium such as a semiconductor memory such as a flash memory, or may have a function for connecting to a reader / writer having a function for reading / writing such a recording medium. Furthermore, the input / output interface 500 may have a function for connecting to other devices.

[0042] The program memory 200 is a non-transient tangible computer-readable storage medium that is a combination of a non-volatile memory that can be written and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a ROM. The program memory 200 stores a program, such as a scenario test program, required for the processor 100 to execute various control processes according to an embodiment. That is, the processing function units in each of the initial value setting unit 2, the simulation unit 3, the search unit 4, and the judgment unit 5 can all be realized by having the processor 100 read and execute the scenario test program stored in the program memory 200. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0043] The data memory 300 is a tangible computer-readable storage medium, for example, a combination of the above-mentioned non-volatile memory and a volatile memory such as a RAM (Random Access Memory). The data memory 300 is used to store various data acquired and created in the course of various processes. That is, the data memory 300 has areas secured for storing various data as appropriate in the course of various processes. As such areas, the data memory 300 may be provided with, for example, a setting storage unit 301, a learning result storage unit 302, a search test result storage unit 304, an evaluation list storage unit 303, and a temporary storage unit 305.

[0044] The setting storage unit 301 stores the initial values ​​of various parameters input by the test engineer TENG when the processor 100 executes the operation as the above-mentioned initial value setting unit 2. That is, when the test engineer TENG inputs the initial values ​​of various parameters from the input unit 700 via the input / output interface 500 on the setting screen displayed on the display unit 800 via the input / output interface 500, the setting storage unit 301 stores the input initial values.

[0045] The learning result storage unit 302 stores the trained policy network and the inverted policy network when the processor 100 executes the operation of the above-mentioned mimicking unit 3.

[0046] The evaluation list storage unit 303 stores the evaluation list when the processor 100 executes the operations of the search unit 4 and the determination unit 5 described above.

[0047] The search test result storage unit 304 stores the search test results when the processor 100 executes the operations of the search unit 4 and the determination unit 5 described above.

[0048] The temporary storage unit 305 stores various intermediate data that is generated during the operation of the processor 100 and is not stored in the setting storage unit 301 , the learning result storage unit 302 , the search test result storage unit 304 , and the evaluation list storage unit 303 .

[0049] (operation) Next, the operation of the scenario test device 1 will be described.

[0050] 5 is a flowchart showing an example of the processing operation of the scenario test device 1. The processor 100 of the engineer terminal TER starts the operation as the scenario test device 1 shown in this flowchart by reading and executing a scenario test program stored in the program memory 200. Execution of the scenario test program in the processor 100 starts when an instruction to carry out a scenario test is received from the input unit 700 via the input / output interface 500, or from an external data processing device via the communication interface 400.

[0051] First, the processor 100 performs the operation as the initial value setting unit 2 to set initial values ​​(step S11). For example, the processor 100 sets, as initial values, the screen number of each screen of the target system SYS, the event number of each screen operation event, the defect judgment criteria, the end condition (e.g., the depth of MCTS) and reward in the search test, and the end condition of the scenario test, according to the input from the input unit 700 via the input / output interface 500. The processor 100 stores each set initial value in the setting storage unit 301 of the data memory 300.

[0052] Here, the setting of screen numbers and event numbers will be explained. Figure 6 is a schematic diagram showing an example of the transition relationship between screens provided by the target system SYS, and Figure 7 is a schematic diagram showing an example of the numbers given to the screens and operation events within the screens in the example of Figure 6. Here, for simplicity, only two screens are shown.

[0053] 6, for example, selection buttons 911 to 914, which are screen elements corresponding to four operation events, are arranged on a first screen 91, and selection buttons 921 and 922 for two operation events are arranged on a second screen 92. When a fourth selection button 914 on the first screen 91 is operated, the target system SYS transitions from the first screen 91 to the second screen 92, and when a second selection button 922 on the second screen 92 is operated, the target system SYS transitions from the second screen 92 to the first screen 91.

[0054] The test engineer TENG knows all of the transition relationships between each screen and the operation events on each screen, and uses this knowledge to assign a screen number as an identifier to each screen and each operation event.

[0055] For example, the test engineer TENG assigns a unique screen number to each screen based on the transition relationship of each screen. In the example of Fig. 6, the test engineer TENG can assign screen numbers "1" to the first screen 91 and "2" to the second screen 92, as shown in Fig. 7. Then, the test engineer TENG inputs the assigned screen numbers for each screen from the input unit 700. In response to this, the processor 100 executes the operation as the screen number setting unit 21 and stores the input screen numbers in the setting storage unit 301.

[0056] In addition, the test engineer TENG can combine the screen numbers given to the screens according to the positions (e.g., up>down, left>right) of the screen elements (selection buttons) corresponding to the operation events on each screen to set the event numbers as identifiers of the operation events. In the example of FIG. 6, the test engineer TENG can issue event numbers "1-1" to "1-4" to the selection buttons 911 to 914 of the first screen 91 and "2-1" and "2-2" to the selection buttons 921 and 922 of the second screen 92, as shown in FIG. 7. Then, the test engineer TENG inputs the event numbers issued for each operation event on each screen from the input unit 700. In response to this, the processor 100 executes the operation as the event number setting unit 22 and stores the input event numbers in the setting storage unit 301.

[0057] Furthermore, the processor 100 receives the fault judgment criteria input by the test engineer TENG via the input unit 700, executes the operation as the judgment criteria setting unit 23, and stores the input fault judgment criteria in the setting storage unit 301. The fault judgment criteria are judgment criteria for determining the part that leads to a certain state (fault), such as when the target system SYS displays a prepared error message, when the target system SYS ends in a state other than the set end condition, etc.

[0058] In addition, the processor 100 receives the search test termination condition and reward input by the test engineer TENG from the input unit 700, executes the operation as the search test setting unit 24, and stores the input search test termination condition and reward in the setting storage unit 301. The search test termination condition is a condition for terminating the search test, which is a trial to search for defects by performing operations and screen transitions in MCTS. For example, two screen transitions, 5 minutes elapsed, etc. can be set as the search test termination condition. In addition, the reward is a size (score) determined by the result of searching until the set search test termination condition is satisfied in the search test, that is, whether or not a defect is found. Note that all states and operation scores of the searched route are managed as an evaluation list, as described later, and stored in the evaluation list storage unit 303. For example, all routes where a defect is found are given a "+1" and the evaluation list is updated. All routes where a defect cannot be found are given a "-1" and the evaluation list is updated. These "+1" and "-1" are set as the reward.

[0059] Furthermore, the processor 100 receives the end condition of the scenario test input by the test engineer TENG via the input unit 700, executes the operation as the scenario test setting unit 25, and stores the input end condition in the setting storage unit 301. An example of the end condition of the scenario test is to end the scenario test after a certain period of time has elapsed.

[0060] Next, the processor 100 executes the operation as the history learning unit 31 of the mimicry unit 3 to learn the policy network by utilizing behavior cloning (step S12). That is, the processor 100 trains the policy network by behavior cloning using multiple scenario test histories including test histories from multiple test engineers TENG (people who often perform scenario tests) and / or scenario histories such as standard scenarios thought up by developers as training data. When a certain screen of the target system SYS is input as a state, the policy network trained in this way can output the probability of selecting as a person for all operation events that can be operated from that state.

[0061] Furthermore, the processor 100 executes the operation as the reversing unit 32 of the mimicking unit 3 to reverse the learned policy network (step S13). That is, the processor 100 reverses the trained policy network, and increases the probability of the least likely selection among the selections for the operation event of the user who operates the target system SYS. For example, by reversing the magnitude relationship of the probability set of the selections for operating as a person obtained in the above step S12, the probability of the selections that are least likely to be operated as a normal person is increased, and the probability of the selections that are most likely to be operated is decreased. In this way, the policy network for MCTS search is trained using behavior cloning, which is a method for learning a model for mimicking human behavior, and the result is reversed, so that the behavior that is least likely to be selected by humans can be imitated. Then, the processor 100 stores this reversed policy network in the learning result storage unit 302.

[0062] 8 is a schematic diagram for explaining an example of the operation of the emulation unit 3. Here, π is a policy function (policy), p is a policy network (policy network), s is a screen state (state), a is an operation (action), and a *Let us define,as the actual action.,The update of the policy network at time,t,is as follows:

[0063] The processor 100 is in state s t With input, n operations a are generated from the policy function π. t (hereafter called action) t Obtain a policy network p that can output

[0064]

number

[0065] where θ is a parameter of the neural network. Actual Action

[0066]

number

[0067] In this case, the processor 100 performs one-hot encoding to obtain the empirical data set, vector y t ∈{0,1} n get.

[0068] Vector p t and vector y t The closer the two are, the closer the behavior is to human behavior. Therefore, the processor 100 uses the gradient descent method to update the policy network.

[0069] The processor 100 then subtracts the results of training the policy network in this manner from an n-dimensional vector whose components are all "1" and inverts the magnitude relationship to produce an inverted policy network of the trained policy network.

[0070]

number

[0071] It is of course possible to use a reversal method other than the one described here.

[0072] That is, the processor 100 reduces the probability of an operational scenario that is most likely to be operated by the user, and increases the probability of an operational scenario that is least likely to be operated by the user.

[0073] Next, the processor 100 operates as the MCTS search unit 41 of the search unit 4 to perform a search test on the target system SYS (step S14). Then, the processor 100 operates as the result judgment unit 51 of the judgment unit 5 to judge whether the result of the search test matches the malfunction judgment condition set by the judgment criterion setting unit 23, that is, whether a malfunction has been found (step S15).

[0074] Here, if it is determined that the search test result does not match the defect judgment condition (NO in step S15), the processor 100 updates the evaluation list with the set reward (step S16). That is, the processor 100 executes the operation as the result judgment unit 51 to set the reward. Then, the processor 100 executes the operation as the update unit 42 of the search unit 4 to update the evaluation list stored in the evaluation list storage unit 303 with the set reward. The processor 100 also executes the operation as the recording unit 43 of the search unit 4 to record the search test result in the search test result storage unit 304 (step S17).

[0075] On the other hand, if it is determined that the search test result meets the defect judgment condition (YES in step S15), the processor 100 notifies the test engineer TENG of the detected defect via the display unit 800 (step S18). At this time, the processor 100 can notify the location where the defect occurred and the content of the defect. After that, the processor 100 proceeds to the processing of the above step S16, updates the evaluation list with the set reward, and further records the search test result in the search test result storage unit 304 in the above step S17.

[0076] After that, the processor 100 executes the operation as the search end judgment unit 52 of the judgment unit 5, and judges whether or not the search test end condition set by the search test setting unit 24 is met, that is, whether or not the search test end condition is met (step S19). If it is judged that the search test end condition is not met (NO in step S19), the processor 100 proceeds to the process of step S14.

[0077] In this way, the processor 100 uses the MCTS to perform a defect search test until the end condition of the set search test is met, and searches for the part and defect that are least likely to be operated by a human obtained from the simulator 3. If a defect is found, the processor 100 notifies the user of the defect and updates the evaluation list and records the search test. If a defect is not found, the processor 100 updates the evaluation list and records the search test.

[0078] FIG. 9 is a schematic diagram for explaining an example of the operation of the search unit 4 and the determination unit 5. As shown in FIG.

[0079] Selection Initial state s at time t t That is, the processor 100 starts with the policy network p t * is used to set the initial state s set by the test engineer TENG using the initial value setting unit 2. t Action a on (root node) t Select.

[0080] In state s, to choose action a, there is score(a) to evaluate.

[0081]

number

[0082] Here, μ is a parameter. Q(a) records the evaluations of all actions of the target system SYS, which is the evaluation list. N(a) is the number of times that the search test for the action is performed.

[0083] In other words, the exploratory test is first performed on a trained policy network p t However, as the number of times N(a) increases, the judgment will be based on evaluation list Q(a).

[0084] Expansion The processor 100 selects the action a t The state transition is from state st to the next state st+1. Initially, the processor 100 randomly samples the next possible action.

[0085]

number

[0086] Conduct exploration tests.

[0087] "Evaluation" When the processor 100 terminates the search test based on the defect determination criteria and the termination conditions of the search test determined by the initial value setting unit 2, it updates the evaluation list using the reward R (for example, "+1" for all operation scenarios that discovered a defect and "-1" for all operation scenarios that did not discover a defect).

[0088] For example, the processor 100 may t A search test is performed from time T to time T.

[0089]

number

[0090] teeth,

[0091]

number

[0092] If you find any defects by

[0093]

number

[0094] If you don't find any defects,

[0095]

number

[0096] Let us assume that.

[0097] [Update (Backpropagation)] The processor 100 reflects all rewards R obtained for each action a in the evaluation Q(a) of that action.

[0098] After performing multiple search tests, the processor 100 can update the evaluation list Q(a) and the number of times N(a) to perform the search tests from the reward R. Since the processor 100 can also obtain the evaluation list Q(a) for the current state s, the processor 100 can use this evaluation list Q(a) to determine whether to continue the search test. For example, if Q(s) is very small, there is no need to perform the search test.

[0099] When it is determined that the end condition of the search test is met (YES in step S19), the processor 100 transitions to the next state of the scenario test based on the result of the search test (step S20). That is, the processor 100 sets the next scenario test to be performed by selecting the most frequently performed action in the search test as the action for the initial state.

[0100]

number

[0101] Then, the processor 100 executes the operation as the scenario end judgment unit 53 of the judgment unit 5, and judges whether or not the scenario test set in the above step S20 matches the end condition of the scenario test (step S21). That is, the processor 100 judges whether or not the set scenario test is the end condition. If it is judged that the end condition of the scenario test is not matched (NO in step S21), the processor 100 proceeds to the process of the above step S14, and repeats the search test.

[0102] Then, when it is determined that the end condition of the scenario test is met (YES in step S21), the processor 100 ends the processing operation of the flowchart shown in Fig. 5. At this time, the processor 100 can notify the test engineer TENG that the scenario test has ended by the display unit 800. Furthermore, the evaluation list stored in the evaluation list storage unit 303 and / or the search test results stored in the search test result storage unit 304 may also be presented to the test engineer TENG by the display unit 800. Alternatively, in response to a request operation from the input unit 700 of the test engineer TENG, the processor 100 may display the evaluation list and / or the search test results on the display unit 800.

[0103] By following the above procedure, it is possible to efficiently perform scenario testing of the target system SYS by conducting exploratory testing while selecting operations that are likely to contain defects and operations that are unlikely to be selected by a human.

[0104] The scenario test device 1 according to the embodiment described above is a scenario test device 1 that performs a scenario test on the target system SYS, and includes an imitation unit 3 and a search unit 4. The imitation unit 3 learns the selection for the operation event of the user who operates the target system SYS, using a plurality of scenario test histories as training data. The search unit 4 performs a search test, which is a trial to search for the defect by performing the operation and screen transition, by giving a higher priority to the operation with a lower probability among the learned selections, or by giving a higher evaluation to the operation in the operation scenario in which a defect was found. Thus, it is possible to confirm the defect that exists in the user operation (unexpected scenario test) that is unlikely to be performed by a human being on the target system SYS. In addition, it is also possible to test the entire target system SYS in a shortened time by testing the operation scenario that is most likely to have a defect.

[0105] Therefore, the scenario testing device 1 according to an embodiment can efficiently discover system malfunctions caused by user operations that the test engineer TENG did not anticipate, regardless of human experience.

[0106] The mimicking unit 3 includes a history learning unit 31 that uses a plurality of scenario test histories as training data to train a policy network that outputs a probability of selection for an operation event of the target system SYS through history learning using behavior cloning, and a reversing unit 32 that generates an inverted policy network by inverting the policy network trained by the history learning unit 31. In this way, by using behavior cloning that mimics human behavior, it becomes possible to test operation scenarios that are unlikely to be performed by humans on the target system SYS.

[0107] Here, the reversing unit 32 generates an inverted policy network by subtracting the trained policy network from an n-dimensional vector whose components are all "1" and reversing the magnitude relationship. Thus, the policy network can be easily inverted.

[0108] The search unit 4 also includes an MCTS search unit 41 that performs search testing using an MCTS, and the MCTS search unit 41 selects an action for the initial state of the MCTS using the inverted policy network from the simulation unit 3. Therefore, operation scenarios that are likely to have defects can be efficiently tested using the MCTS based on the training results of the simulation unit 3.

[0109] Moreover, the scenario test device 1 according to an embodiment further includes a search test setting unit 24 that sets an end condition of the search test, and a search end judgment unit 52 that judges whether the search test by the MCTS search unit 41 has reached the end condition of the search test, and when the search end judgment unit 52 judges that the search test has reached the end condition of the search test, the MCTS search unit 41 transitions to the next state of the scenario test and repeats the search test. Thus, the end condition of the search test can be set arbitrarily.

[0110] Moreover, the scenario test device 1 according to one embodiment further includes a scenario test setting unit 25 that sets an end condition of the scenario test, and a scenario end judgment unit 53 that judges whether the state next to the transitioned scenario test meets the end condition of the scenario test, and the MCTS search unit 41 ends the scenario test when the scenario end judgment unit 53 judges that the state next to the transitioned scenario test meets the end condition of the scenario test. Thus, the end condition of the scenario test can be set arbitrarily.

[0111] [Other embodiments] In the embodiment, when the initial values ​​are set, a number is assigned as an identifier to each screen and each operation event. However, other identifiers such as character strings may be used.

[0112] Moreover, the scenario test device 1 does not necessarily have to be configured by one computer. For example, the history learning unit 31 of the simulation unit 3 may be configured on a server that can be accessed by multiple engineer terminals TER via a network. In this way, it becomes possible to perform different or the same scenario tests in parallel for one target system SYS by multiple engineer terminals TER using a common policy network.

[0113] The method described in the embodiment can be stored as a program (software means) that can be executed by a calculator (computer) on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium side also includes a setting program that configures the software means (including not only execution programs but also tables and data structures) that the calculator executes within the computer. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium referred to in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0114] In short, this invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. Moreover, each embodiment may be implemented in combination as appropriate as possible, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed constituent elements. [Explanation of symbols]

[0115] 1. Scenario test equipment 2. Initial value setting section 3. Imitation Department 4…Exploration Department 5... Judgment section 21...Screen number setting section 22...Event number setting section 23…Judgment criteria setting section 24…Exploratory test setting section 25…Scenario Test Setting Department 31…History learning section 32…Reverse section 41…MCTS Exploration Department 42…Update section 43…Recording section 51...Result judgment section 52...Search end determination unit 53…Scenario end judgment section 91…First screen 92…Second screen 100…Processor 200…Program memory 300…Data memory 301...Settings memory section 302…Learning result storage unit 303...Evaluation list storage unit 304...Search test result storage unit 305...Temporary storage section 400…Communication interface 500... Input / Output Interface 600…Bus 700…Input section 800...Display section 911~914,921,922...Select button SYS: Target system TENG...Test engineer TER: Engineer terminal

Claims

1. A scenario test device for performing a scenario test on a target system, an imitating unit that learns selections for operation events of a user who operates the target system using a plurality of scenario test histories as training data; a search unit that performs a search test, which is a trial to search for defects by performing operations and screen transitions, by giving a higher priority to operations with lower probability among the learned selections, or by giving a higher evaluation to an operation in an operation scenario in which a defect is found; A scenario test device comprising:

2. The mimicking portion is a learning unit that trains a policy network that outputs a probability of selection for the operation event of the target system by history learning using behavior cloning, using the multiple scenario test histories as training data; an inversion unit for generating an inverted policy network by inverting the trained policy network; The scenario test device according to claim 1 , comprising:

3. 3. The scenario testing device according to claim 2, wherein the reversing unit generates the inverted policy network by subtracting the trained policy network from an n-dimensional vector whose components are all "1" and reversing the magnitude relationship.

4. the search unit includes an MCTS search unit that performs the search test by a Monte Carlo tree search; 4. The scenario testing apparatus of claim 2, wherein the MCTS search unit selects an action for an initial state of the Monte Carlo tree search using the inverted policy network.

5. A search test setting unit that sets an end condition of the search test; a search end determination unit that determines whether the search test by the search unit has reached an end condition of the search test; Further comprising:

5. A scenario testing device according to claim 1, wherein the search unit, when the search termination determination unit determines that the search test has reached a termination condition for the search test, transitions to a next state of the scenario test and repeats the search test.

6. a scenario test setting unit that sets a termination condition of the scenario test; a scenario end determination unit that determines whether a next state of the transitioned scenario test satisfies an end condition of the scenario test; Further comprising:

6. The scenario testing device according to claim 5, wherein the search unit terminates the scenario test when the scenario termination determination unit determines that the state next to the transitioned scenario test satisfies a termination condition for the scenario test.

7. A scenario testing method in a scenario testing device having a processor and performing a scenario test on a target system, comprising: The processor learns selections for operation events of a user who operates the target system using a plurality of scenario test histories as training data; the processor assigns a higher priority to an operation with a lower probability among the learned selections, or assigns a higher evaluation to an operation in an operational scenario in which a defect is found, and performs a search test, which is a trial to search for the defect by performing the operation and screen transitions; Scenario testing methodology.

8. 7. A scenario test program for causing a processor to function as each of the components of the scenario test device according to claim 1.

Images