Program for controlling order to execute multiple test sequences, control device and control method

Abstract

To control an order in which multiple test sequences are executed.SOLUTION: A control device 101 controls an order in which multiple test sequences are executed. A detection unit 111 detects that a test has failed while a test sequence is being executed. If no defect candidates have yet been set, a setting unit 112 sets a functional unit used in the failed test to a defect candidate. If defect candidates have already been set, a narrowing unit 113 narrows down the defect candidates to a functional unit used in the failed test. If there is one functional unit with a defect candidate, an output unit 114 outputs a message indicating that the one functional unit has a defect. If there are multiple functional units with defect candidates, a selection unit 115 selects, as a test sequence to be executed next, a test sequence in which the functional units used in the test defined by the test sequence narrow down the multiple functional units that have the defect candidates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, a control device, and a control method for controlling the order in which a plurality of test sequences are executed. [Background technology]

[0002] Devices that provide various services, such as web services, are realized by executing programs associated with those services, and in providing those services, numerous tests are periodically and intermittently executed to confirm that there are no defects. If all tests are completed successfully, it means that no defects have been found, but if any test fails, it means that there is a defect in one of the functional units executed by that test.

[0003] Some tests have dependencies on the order in which they are executed. Therefore, one or more tests arranged in the order in which they are executed according to their dependencies is called a test sequence. In other words, a test sequence determines the order in which one or more tests are executed.

[0004] When S test sequences 1, 2, ..., S are prepared as one test mission, the test mission is executed periodically and intermittently, for example, by batch processing. To enable the execution order of test sequences 1, 2, ..., S to be changed arbitrarily within the test mission, tests that have dependencies on each other in the execution order are grouped into one test sequence, as described above.

[0005] Here, let us assume that the number of tests whose execution order is determined by each test sequence s∈{1, 2, ..., S} is N(s). The N(s) tests whose execution order is determined by test sequence s are denoted as v(s,1), v(s,2), ..., v(s,N(s)).

[0006] The tth test v(s,t) executed in the test sequence s uses one or more functional units in its execution to check whether the one or more functional units have any defects. Here, one of the functional units may be one of the following, as appropriate: (1) One middleware. The middleware consists of one or more modules / packages. (2) A module / package, which consists of one or more source files. (3) One source file, which consists of one or more functions. (4) A function, which references one or more resource files. (5) One resource file. (6) A combination of these. For example, some middleware A may treat the whole as one functional unit, while other middleware B may treat each of multiple source files as a functional unit.

[0007] Below, the set of functional units used in test v(s,t) is defined as m(s,t) = {m s,t,1 , m s,t,2 , ..., m s,t,M(s,t)} is marked as

[0008] Patent Document 1 discloses a technology that extracts attributes of the deliverables that were the subject of the changed test during the period when the test result changed from success to failure or from failure to success, and narrows down related candidates when associating the defect with other deliverables related to the defect. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-69615 Summary of the Invention [Problem to be solved by the invention]

[0010] When a test v(s,t) that is defined as the tth test to be executed in a test sequence s is executed and fails, the M(s,t) functional units m(s,t) included in the set m(s,t) are executed. s,t,1 , m s,t,2 , ..., m s,t,M(s,t) There is probably a problem with one of them.

[0011] Here, M(s,t)=1, that is, if the number of elements #m(s,t) of the set m(s,t) is 1, the defective functional unit is m s,t,1 It is confirmed that this is the case.

[0012] If M(s,t)>1, that is, the number of elements in the set m(s,t) is more than one, then #m(s,t) functional units m s,t,1 , m s,t,2 , ..., m s,t,M(s,t) There is a demand to narrow down as quickly as possible which of the above has a defect by controlling the order in which the test sequences are executed.

[0013] The present invention is intended to solve the above-mentioned problems, and has as its object to provide a program, a control device, and a control method for controlling the order in which a plurality of test sequences are executed. [Means for solving the problem]

[0014] In order to solve the above problems, a control device according to the present invention is realized by causing a computer to execute a program for controlling an order in which a plurality of test sequences are executed, each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; The control device includes: Detecting that a test has failed during the execution of a test sequence; If it is detected that the test has failed and a defect candidate has not yet been set, set the functional unit used in the failed test as the defect candidate; If it is detected that the test has failed and the defect candidate has already been set, narrowing down the defect candidate functional units to the functional units used in the failed test; If there is one functional unit that is a defect candidate, outputting a message indicating that the one functional unit has a defect; If there are a plurality of functional units that are defect candidates, a selection unit selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, and selects the test sequence to be executed next so that a functional unit used in any test determined by the test sequence to be executed next narrows down the plurality of functional units that are defect candidates. Configure it as follows. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a program, a control device, and a control method for controlling the order in which a plurality of test sequences are executed. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is an explanatory diagram showing a schematic configuration of a control device according to an embodiment of the present invention; [Figure 2] 3 is a flowchart showing a flow of a control process executed by a control device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art may employ embodiments in which each or all of the elements are replaced with equivalents, and these embodiments are also within the scope of the present invention.

[0018] (How the control device is realized) The control device according to the present embodiment is generally realized by causing a computer to execute a control program, in which case the computer may execute a service program for implementing the service to be tested, or may have another computer execute the service program.

[0019] For example, if the service is a web service, a server computer that executes a web service program to provide the web service may itself execute a test program and function as a control device.

[0020] Furthermore, the test program may be executed on a client computer that can receive the web service from the server computer, and the client computer may function as a control device.

[0021] As the client computer, in addition to a general computer, a smartphone or a tablet computer can also be used.

[0022] Furthermore, the control device can be realized by having the processing executed by a dedicated electronic circuit.

[0023] In addition, as an intermediate form between a computer and a dedicated electronic circuit, it is also possible to configure the control device of this embodiment by applying technology such as FPGA (Field Programmable Gate Array), which compiles a test program into a design script for an electronic circuit and dynamically configures the electronic circuit based on the design script.

[0024] This test program can be recorded on a computer-readable non-transitory information recording medium such as a compact disc, flexible disk, hard disk, magneto-optical disk, digital video disk, magnetic tape, ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable ROM), flash memory, semiconductor memory, etc. This information recording medium can also be distributed or sold independently of a computer.

[0025] In a computer, a program recorded on a non-transitory information recording medium such as a flash memory, hard disk, or SSD (Solid State Drive) is read into RAM (Random Access Memory), which is a temporary storage device, and then the CPU (Central Processing Unit) executes the instructions contained in the read program.

[0026] However, in architectures that allow ROM and RAM to be mapped into a single memory space and executed, the CPU directly reads and executes the instructions contained in the program stored in ROM.

[0027] Furthermore, the test program can be distributed to various computers via a transitory transmission medium such as a computer communication network, independently of the computer on which the test program is executed, and executed in different environments.

[0028] (Test example) The relationship between test sequences, tests, and functional units will be explained again below. Tables 1 to 6 show examples of test sequences for web services.

[0029] [Table 1]

[0030] [Table 2]

[0031] [Table 3]

[0032] [Table 4]

[0033] [Table 5]

[0034] [Table 6]

[0035] In this example, six test sequences are shown, and tests are executed for eight functional units: "Apache", "PHP", "JavaScript in CDN", "DB", "Internal API", "3rd Party payment API", "3rd Party Login page", and "Search Engine". Therefore, S=6.

[0036] where: In Test Sequence 1 "Purchase Item" according to Table 1, six tests corresponding to steps 1-6 are executed in sequence. Therefore, N(1)=6. In test sequence 2 "My Page" according to Table 2, three tests corresponding to steps 1-3 are executed in sequence. Therefore, N(2)=3. In Test Sequence 3 "Change Payment" according to Table 3, three tests corresponding to steps 1-3 are executed in sequence. Therefore, N(3)=3. In test sequence 4 "Open Native App and Login" according to Table 4, two tests corresponding to steps 1 and 2 are executed in sequence. Therefore, N(4)=3. In test sequence 5 "Favorite" according to Table 5, three tests corresponding to steps 1-3 are executed in sequence. Therefore, N(5)=3. In test sequence 6 "Campaign" according to Table 6, one test corresponding to step 1 is executed in sequence. Therefore, N(6)=1.

[0037] The functional unit used in each test in each table is indicated by a "√" symbol (check mark). For example, The set of functional units "Apache", "PHP", "JavaScript in CDN", "DB", and "Internal API" used in test v(1,4) in step 4 "Choose Item" of test sequence 1 "Purchase Item" is m(1,4) = {1, 2, 3, 4, 5}. The set of functional units "Apache" and "PHP" used in test v(2,1) in step 1 "Go to Top MyPage" of test sequence 2 "My Page" is m(2,1) = {1, 2}. The set of functional units "3rd Party Login page" used in test v(3,2) in step 2 "Login" of test sequence 3 "Change Payment" is m(3,2) = {7}. The set of functional units "Internal API" and "3rd Party Login page" used in test v(4,2) in step 2 "Login" of test sequence 4 "Open Native App and Login" is m(3,2) = {5, 7}. The set of functional units "Internal API", "3rd Party Login page", "JavaScript in CDN", and "Search Engine" used in test v(5,2) in step 2 "Login" of test sequence 5 "Favorite" is m(3,2) = {1, 2, 3, 8}. The set of functional units "Internal API" and "JavaScript in CDN" used in test v(6,1) in step 1 "Go Campaign" of test sequence 6 "Campaign Page" is m(3,2) = {1, 3}.

[0038] When the control device according to this embodiment is applied to the above example, the control device controls the order in which the six test sequences are executed. If all tests in all test sequences are successful, there is no problem. However, if a test in a test sequence fails, the control device narrows down which functional unit has the problem as quickly as possible and outputs one functional unit that has the problem.

[0039] For example, when test sequence 3 "Change Payment" is executed, the functional unit used by test v(3,1) related to step 1 "Go to Payment" is m(3,1) = {1, 2}, and if this test fails, it is considered that there is a defect in either functional unit 1 "Apache" or functional unit 2 "PHP." In other words, the defect candidate at this point is F = {1, 2}.

[0040] So, to narrow this down, Tests that use "Apache" but not "PHP", or Tests that do not use "Apache" but use "PHP" You just need to do one of the following.

[0041] In the above example, the only test that uses "Apache" but not "PHP" is test v(6,1) in test sequence 6. Therefore, the control device selects test sequence 6, which defines test v(6,1), as the next test to be executed.

[0042] If the selected test sequence 6 is executed and the test v(6,1) fails, the current fault candidate F is narrowed down by the set m(6,1) of functional units used in the test v(6,1). That is, F∩m(6,1) = {1, 2}∩{1} = {1} As shown above, the search can be narrowed down using the set operation "∩".

[0043] On the other hand, if test v(6,1) does not fail, the fault candidate F cannot be narrowed down any further using test sequences 1 to 6 in this example. Therefore, the fault candidate at that point is output along with a message that the narrowing down was not possible any further. After this, any unexecuted test sequences may not be executed, or any unexecuted test sequences may be executed to search for faults in functional units other than the fault candidates that have already been output.

[0044] The basic function of the control device according to this embodiment for narrowing down defect candidates has been described above. The control device according to this embodiment will now be described in more detail.

[0045] The above example is provided to facilitate understanding of this embodiment, and the number of test sequences S used in actual operation is usually much larger. Also, the number of tests N(s) executed in a given test sequence s can be changed as desired.

[0046] (Overview of the control device) 1 is an explanatory diagram showing the general configuration of a control device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0047] The control device 101 according to this embodiment executes a program to control the order in which a plurality of test sequences are executed.

[0048] As described above, each test sequence of the plurality of test sequences defines an order in which one or more tests are executed.

[0049] Additionally, one or more functional units are utilized in the execution of each test of the one or more tests.

[0050] Here, the control device 101 includes a detection unit 111, a setting unit 112, a narrowing down unit 113, an output unit 114, and a selection unit 115. In addition, an extraction unit 116 may be further included as an optional element.

[0051] The detection unit 111 detects that a certain test has failed during the execution of a certain test sequence.

[0052] In accordance with the above example, it is assumed that while a plurality of test sequences are being executed in sequence, the detection unit 111 detects that test v(3,1) of test sequence 3 has failed.

[0053] Then, if it is detected that the test has failed and the defect candidate F has not yet been set, the setting unit 112 sets the functional unit used in the failed test as the defect candidate F.

[0054] In the above example, since no test has failed up to that point, the defect candidate F is an empty set (φ), which means that the defect candidate F has not yet been set. Therefore, the setting unit 112 sets the set m(3,2} = {1,2} of functional units as the defect candidate F.

[0055] Now, if there are multiple functional units that are defect candidates, the selection unit 115 selects the next test sequence to be executed from the test sequences that have not yet been executed among the multiple test sequences, so that the multiple functional units that are defect candidates are narrowed down to a functional unit used in any of the tests defined by the next test sequence to be executed.

[0056] In accordance with the above example, since the defect candidates F={1, 2}, the number of functional units that are defect candidates is #F=2, and the selection unit 115 selects A test v(s,t) that has not yet been executed, The functional unit m(s,t) used by the test v(s,t) narrows down the defect candidates F. A test sequence s such that:

[0057] where: F∩m(s,t)⊂F and F∩m(s,t)≠F and F∩m(s,t)≠φ That is, 0 < #〔F∩m(s,t)〕< #F If the above condition is satisfied, the functional unit m(s, t) narrows down the defect candidates F.

[0058] In the above example, only test v(6,1) can narrow down the fault candidates F={1,2}, so test sequence 6 is selected to be executed next.

[0059] Then, if it is detected that the test has failed and defect candidates have already been set, the narrowing-down unit 113 narrows down the functional units that are defect candidates to the functional units used in the failed test.

[0060] In the above example, test sequence 6, which is selected as the next sequence to be executed, is executed, and the fact that test v(6,1) has failed is detected by the detection unit 111.

[0061] At this point, since the defect candidate F≠φ, the narrowing-down unit 113 narrows down the defect candidate F by the calculation F ← F∩m(6,1) In this example, before the test v(6,1) was executed, F={1,2}, so the failure of the test v(6,1) resulted in F ← {1,2}∩{1} = {1} Therefore, the defect candidate F={1}, and the number of functional units included in it, #F, is 1.

[0062] Finally, if there is one functional unit that is a defect candidate, the output section 114 outputs a message indicating that the one functional unit has a defect.

[0063] In the above example, the defect candidate F={1}, and there is one functional unit that is a defect candidate, so the output unit 114 outputs that functional unit 1 has a defect.

[0064] As described above, the defect candidate F corresponds to a set of IDs representing functional units. Therefore, in this embodiment, a candidate set representing the defect candidate F is stored in a storage area 131 such as a RAM or SSD of a computer. For ease of understanding, the symbol "F" used for the defect candidate F is also used for the candidate set.

[0065] If the candidate set F is stored in a storage area 131 such as a RAM or SSD using data representations such as an array, a bit array, an associative array, a vector, a queue, or a stack, it becomes possible to set or narrow down the elements of the candidate set F.

[0066] As described above, if it is detected that the test v(s, t) has failed and the candidate set F is an empty set φ, the setting unit 112 F ← m(s,t) The functional units (elements of m(s,t)) used in the failed test v(s,t) are added to the candidate set F by

[0067] Furthermore, if it is detected that the test v(s, t) has failed and the candidate set F is not an empty set φ, the narrowing unit 113 F ← F∩m(s,t) From the candidate set F, remove functional units that are not utilized in the failed test v(s,t).

[0068] If the candidate set F has one element, the output unit 114 outputs a message indicating that the functional unit that is the one element has a defect.

[0069] If all of the multiple test sequences are executed and there are multiple functional units that are defect candidates, the output section 114 outputs a message indicating that at least one of the multiple defect candidate functional units has a defect.

[0070] In the above example, when all test sequences have been executed, if the number of elements in candidate set F is #F≧2, each element of candidate set F is output, and it is reported that there is a defect in one or more functional units, but it has not been possible to narrow it down to "one functional unit that is definitely defective."

[0071] Also, instead of or in addition to "when all test sequences have finished running," a limit based on the amount of time that has elapsed since the start of test sequence execution may be imposed. For example, if test sequences 1, 2, ..., S are executed daily to monitor the operational status of a service, once a malfunction occurs, the time required to execute all test sequences tends to be longer than when there are no malfunctions, due to factors such as the need to wait for timeouts in various processes.

[0072] Therefore, if the total time required to execute the executed test sequences among the multiple test sequences exceeds a predetermined time and there are multiple functional units that are defect candidates, the output unit 114 may output a message indicating that there is a defect in at least one of the multiple defect candidate functional units.

[0073] This "predetermined time" may be determined, for example, by the average total time required to execute all test sequences when no defects are found in all test sequences, or may be determined according to the frequency with which all test sequences are repeatedly executed. For example, if all test sequences are executed every day and the required time is approximately H hours, the predetermined time may be set to k × H hours, which is a constant k times that time.

[0074] If all of the multiple test sequences are executed without detecting any failure, the output section 114 outputs a message indicating that there was no malfunction in any of the one or more functional units.

[0075] In this way, in this embodiment, attention is focused on the functional units included in the candidate set F to narrow down the functional units that have a defect, but depending on the functional units used by the combination of multiple failed tests, it may be possible to narrow down the functional units to those that are not included in the candidate set F. The extraction unit 116 extracts such combinations.

[0076] That is, the extraction unit 116 extracts, from a plurality of combinations of a failure set whose elements are tests detected as having failed by the detection unit 111, A combination in which one functional unit is used in all tests included in the combination Attempt to extract.

[0077] For example, in a mode in which the history of tests that have been detected as having failed is stored in the storage area 131 as a failure set R, when the number of tests included in the failure set R is #R, the combinations for the failure set can be determined by which of the #R tests to select and which not to select. That is, each combination corresponds to a subset of the "set having #R elements." Therefore, the combinations can be determined by a total of 2 #R There is a street.

[0078] In other words, the combinations here are the elements of the power set Power(R) of the failure set R, and the number of elements of the power set Power(R) is 2#R , and all elements of the power set Power(R) are included in the failure set R.

[0079] So, 2 #R For each combination of the possible combinations, check the functional units used in all the tests included in that combination. W∈Power(R) For a combination W that satisfies the above (i.e., a subset of the failure set R and an element of the power set Power(R) of the failure set R), X = ∩ (s,t)∈W m(s,t) Thus, a set X of functional units used in all tests included in the combination can be obtained.

[0080] And if "the number of functional units used in all tests included in the combination is one", that is, #X = #〔∩ (s,t)∈W m(s,t) = 1 If so, the extraction unit 116 extracts the combination W.

[0081] On the other hand, if a combination is not found in which "the number of functional units used in all tests included in the combination is one", the attempt by the extraction unit 116 will fail.

[0082] If the extraction by the extraction unit 116 is successful, the output unit 114 outputs one functional unit related to the extracted combination W, that is, [∩ (s,t)∈W m(s,t) = {u} It outputs that one functional unit u included in has a defect.

[0083] In the following, various methods for selecting a test sequence s, a test v(s, t), and a set m(s, t) of functional units used by the test, which narrow down the candidate set F, will be described.

[0084] The first method is to select the unexecuted test sequences. 0 < #〔F∩m(s,t)〕< #F In this method, the order of the test sequences 1, 2, ..., S that are executed periodically is not changed, but some of them are skipped to narrow down the faulty functional unit.

[0085] The second method is to select the unexecuted test sequences. 0 < #〔F∩m(s,t)〕< #F Then, a test sequence s for the selected s is selected. This method involves changing the execution order of test sequences that are executed after a defect is discovered.

[0086] The third method uses the expected time required for each test. In an environment where test sequences 1, 2, ..., S are executed periodically, the expected time e(s,t) required for each test v(s,t) can be estimated by collecting and analyzing the execution history.

[0087] Then, the expected time from the start of execution of test sequence s until the end of test v(s,t) related to step t in test sequence s is E(s,t) = Σ i=1 t e(s,i) It can be calculated as follows:

[0088] So, for unexecuted test sequences, 0 < #〔F∩m(s,t)〕< #F Among s that satisfy E(s,t) = Σ i=1 t e(s,i) The test sequence s for s that has the shortest length is selected as the next test to be executed. This method speeds up the narrowing down process by executing the narrowed down tests as early as possible.

[0089] The fourth method is to select the test sequence that narrows down the problem as much as possible when a defect is found. That is, for the unexecuted test sequence, 0 < #〔F∩m(s,t)〕< #F Among the s that satisfy the above, the test sequence s for s that has the smallest #[F∩m(s,t)] is selected as the next one to be executed. This method speeds up the narrowing down process by maximizing the degree of narrowing down.

[0090] The fifth method is to combine the above methods as appropriate. For example, (1) In the fourth method, if there are multiple cases where #[F∩m(s,t)] has the same minimum value, select the one with the shortest E(s,t) from the multiple cases using the third method. (2) Randomly select a predetermined number of E(s,t) from the top of the list sorted in order of shortest length, or (3) Select the one with the smallest evaluation value calculated based on E(s,t) and #[F∩m(s,t)] (for example, the score calculated by multiplying each by a weight). In addition to any of the above three methods, various other combinations can be adopted.

[0091] (control processing) 2 is a flowchart showing the flow of control processing executed by a control device according to an embodiment of the present invention. The following description will be made with reference to this figure. Note that each step described below can be modified or omitted as appropriate depending on the application.

[0092] When this process starts, the control device 101 initializes the candidate set F to an empty set φ (step S201), and initializes the failure set R to an empty set φ (step S202).

[0093] Then, if there are any unexecuted test sequences among the S test sequences 1, 2, ..., S related to the service to be tested (step S203; Yes), the control device 101 selects one of the unexecuted test sequences s as the next to be executed in a predetermined order or randomly (step S204).

[0094] The default order is: in a predetermined order, In order of frequency of past failures, The sequences used in the actual operation of the service are assigned to one of the test sequences, and then ranked in order of the number of times they were assigned. A combination of these Various orders can be adopted.

[0095] Furthermore, the control device 101 executes the selected test sequence s by itself or causes the server computer that provides the service to execute it (step S205).

[0096] If the test sequence s is successful (step S206; successful), the control device 101 advances the process to step S241.

[0097] On the other hand, if it is detected that the test sequence s has failed (step S206; failure), the control device 101 identifies the failed test v(s, t) (step S207).

[0098] If the candidate set F is an empty set φ (step S208; empty set), the control device 101 adds a set m(s, t) of functional units used by the test v(s, t) to the candidate set F. F ← m(s,t) (step S209), and the control proceeds to step S221.

[0099] If the candidate set F is not an empty set, that is, if the candidate set F has already been set (step S208; set), the control device 101 checks whether the intersection F∩m(s,t) between the candidate set F and the set m(s,t) of functional units used by the test v(s,t) is an empty set φ (step S210).

[0100] If φ is an empty set (step S210;=φ), the control device 101 advances the process to step S211.

[0101] If the candidate set F is not an empty set φ (step S210; ≠ φ), the control device 101 defines the candidate set F as follows: F ← F∩m(s,t) (step S211), and the control proceeds to step S221.

[0102] Now, if the candidate set F has one element, that is, F={u} (step S221; Yes), the control device 101 outputs a message indicating that the functional unit u has a defect (step S222), and ends this process.

[0103] On the other hand, if the candidate set F has a plurality of elements, that is, if F≠{u} (step S221; No), the control device 101 advances the process to step S231.

[0104] Now, if the common part F∩m(s, t) is an empty set φ (step S209;=φ), it means that a defect has been found in a functional unit that is not included in the candidate set currently being considered.

[0105] Therefore, the control device 101 generates the identification information (s, t) for identifying the test v(s, t) in which the defect was found, as follows: R←R∪{(s,t)} Then, the following process is repeated for each element W of the power set Power(R) of the failure set R (step S232). As described above, each element W is a subset of the failure set R (W ⊂ R), and corresponds to a combination of elements extracted from the failure set R.

[0106] First, a set X of functional units used by all of the tests v(s, t) identified by the elements (s, t) of the control device 101 subset V is defined as follows: X←∩ (s,t)∈W m(s,t) (step S233).

[0107] Then, it is checked whether the number of elements in the set X is 1, that is, whether X={u}. If X={u} (step S234; Yes), the control device 101 outputs a message indicating that there is a problem with the functional unit u (step S235), and ends this process.

[0108] If X≠{u} (step S234; No), the control device 101 continues the repetition (step S236). When the repetition ends, the control device 101 advances the process to step S241.

[0109] After the above processing is completed, the control device 101 checks whether the candidate set F has been set or is an empty set (step S241).

[0110] If the candidate set F has been set (step S241; set), the control device 101 checks whether a new test sequence s remains that can be narrowed down (step S242). As described above, for the test v(s, t) and the set m(s, t) of functional units, m(s, t) is set up so that m(s, t) narrows down the candidate set F, that is, 0<#〔F∩m(s,t)〕< #F The next test sequence s to be executed is found using the above five examples or a method similar to them so as to satisfy the following.

[0111] If such a new test sequence s is found (step S243: YES), the new test sequence s is selected (step S244), and the control proceeds to step S205.

[0112] On the other hand, if a new test sequence s is not found (step S243; None), the control device 101 outputs a message that there is a defect in one of the elements of the candidate set F, but it has not been possible to identify which one (step S244), and ends this process.

[0113] Also, if F is an empty set (step S241; empty set), the control device 101 advances the process to step S203.

[0114] In addition, if there are no unexecuted test sequences (step S203; None), and if the candidate set F has been set (step S251; Set), the control device 101 outputs the elements of the candidate set F (step S252), as in step S235, and terminates this processing.

[0115] On the other hand, if the candidate set F is an empty set φ (step S251; empty set), the control device 101 outputs a message that no defect has been found (step S253), and ends this process.

[0116] In addition, in the output by the output unit 114 according to this embodiment, If no problems are found, it will output a message to that effect. If the defective functional unit can be narrowed down to one, output that functional unit. If there are multiple defective functional units and it is not possible to narrow them down, output those functional units. However, The history of elements added to the failure set R, i.e., the history of failed tests, History of successful test sequences etc. may also be output together.

[0117] In addition, the output unit 114 Information identifying the test sequences and tests used to narrow the candidate set, i.e. F←m(s,t) or, F←F∩m(s,t) or, X←∩ (s,t)∈W m(s,t) It is also possible to output the history of (s, t) used in

[0118] By examining this information, it is possible to know which test in which test sequence has failed.

[0119] Furthermore, although the above control does not impose a restriction based on the time elapsed since the start of execution of the test sequence, it is possible to impose a restriction based on the elapsed time by, for example, checking whether the elapsed time has exceeded a predetermined upper limit just before executing step S242, and if so, proceeding to step S251.

[0120] In an aspect in which the extraction unit 116 is omitted, the execution of steps S202 and S231-236 related to the failure set R can be omitted.

[0121] As described above, according to this embodiment, by controlling the order in which a plurality of test sequences are executed, it becomes possible to narrow down which functional unit has caused the defect as early as possible.

[0122] This embodiment is particularly suitable when priority is given to detecting one functional unit that is almost certainly defective as early as possible, rather than detecting all functional units that are likely to be defective.

[0123] (summary) The program, control device, and control method according to the present embodiment described above will be described below.

[0124] (Appendix 1) A program for controlling an order in which a plurality of test sequences are executed, each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; The program causes a computer to: a detection unit for detecting a failure of a test during execution of a test sequence; a setting unit that sets a functional unit used in the failed test as a defect candidate if the test has failed and a defect candidate has not yet been set; a narrowing-down unit that narrows down the functional units that are the defect candidates to the functional units used in the failed test, if it is detected that the test has failed and the defect candidates have already been set; an output unit that outputs, if there is one functional unit that is a defect candidate, a message indicating that the one functional unit has a defect; a selection unit that selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, if there are a plurality of functional units that are defect candidates, such that a functional unit used in any test determined by the test sequence to be executed next narrows down the plurality of defect candidate functional units; A program characterized by functioning as

[0125] (Appendix 2) a candidate set representing the defect candidates is stored in a storage area of ​​the computer; If it is detected that the test has failed and the candidate set is an empty set, the setting unit adds a functional unit used in the failed test to the candidate set; If it is detected that the test has failed and the candidate set is not an empty set, the narrowing unit removes from the candidate set functional units that are not used in the failed test; If the candidate set has one element, the output unit outputs a message indicating that the functional unit that is the one element has a defect. 2. The program according to claim 1,

[0126] (Appendix 3) When all of the plurality of test sequences are executed and there are a plurality of functional units that are defect candidates, the output unit outputs a message indicating that at least one of the plurality of defect candidate functional units has a defect. 3. The program according to claim 1 or 2,

[0127] (Appendix 4) If the total time required to execute the executed test sequences among the plurality of test sequences exceeds a predetermined time and there are a plurality of functional units that are defect candidates, the output unit outputs a message indicating that at least one of the plurality of defect candidate functional units has a defect. 4. The program according to any one of appendices 1 to 3.

[0128] (Appendix 5) If all of the plurality of test sequences are executed without detecting any failure, the output unit outputs a message indicating that there is no defect in any of the one or more functional units. 5. The program according to any one of claims 1 to 4.

[0129] (Appendix 6) Among the unexecuted test sequences, a test sequence that defines a test that utilizes a functional unit that narrows down the plurality of functional units that are defect candidates and that has the shortest expected time from the start of execution of the test sequence to the end of the test is selected as the test sequence to be executed next. 6. The program according to any one of appendices 1 to 5.

[0130] (Appendix 7) Among the unexecuted test sequences, a test sequence that defines a test that uses functional units that result in the smallest number of functional units after narrowing down the plurality of functional units that are defect candidates is selected as the test sequence to be executed next. 6. The program according to any one of appendices 1 to 5.

[0131] (Appendix 8) From a plurality of combinations of a failure set whose elements are tests detected as having failed by the detection unit, A combination in which one functional unit is used in all tests included in the combination An extractor that attempts to extract Furthermore, If the extraction is successful, the output unit outputs a message indicating that one functional unit related to the extracted combination has a defect. 8. The program according to any one of appendices 1 to 7,

[0132] (Appendix 9) A control device for controlling an order in which a plurality of test sequences are executed, each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; The control device a detection unit for detecting a failure of a test during execution of a test sequence; a setting unit that sets a functional unit used in the failed test as a defect candidate if the test has failed and a defect candidate has not yet been set; a narrowing-down unit that narrows down the functional units that are the defect candidates to the functional units used in the failed test, if it is detected that the test has failed and the defect candidates have already been set; an output unit that outputs, if there is one functional unit that is a defect candidate, a message indicating that the one functional unit has a defect; a selection unit that selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, if the plurality of functional units are defect candidates, such that the plurality of functional units that are defect candidates are narrowed down to functional units to be used in a test determined by the test sequence to be executed next; A control device comprising:

[0133] (Appendix 10) A control method executed by a control device that controls an order in which a plurality of test sequences are executed, comprising: each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; a detecting step in which the control device detects that a test has failed during execution of a test sequence; a setting step in which, when it is detected that the test has failed and a failure candidate has not yet been set, the control device sets the functional unit used in the failed test as the failure candidate; a narrowing-down process in which, if it is detected that the test has failed and the defect candidates have already been set, the control device narrows down the defect candidate functional units to the functional units used in the failed test; an output step in which, if there is one functional unit that is a defect candidate, the control device outputs an output indicating that the one functional unit has a defect; a selection step in which, if there are a plurality of functional units that are defect candidates, the control device selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, the selection step being performed by the selection unit, such that a functional unit used in any test determined by the test sequence to be executed next narrows down the plurality of defect candidate functional units; A control method comprising:

[0134] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]

[0135] According to the present invention, it is possible to provide a program, a control device, and a control method for controlling the order in which a plurality of test sequences are executed. [Explanation of symbols]

[0136] 101 Control device 111 Detection unit 112 Setting section 113 Narrowing section 114 Output section 115 Selection Section 116 Extraction part

Claims

1. A program for controlling an order in which a plurality of test sequences are executed, each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; The program causes a computer to: a detection unit for detecting a failure of a test during execution of a test sequence; a setting unit that sets a functional unit used in the failed test as a defect candidate if the test failure is detected and a defect candidate has not yet been set; a narrowing-down unit that narrows down the functional units that are the defect candidates to the functional units used in the failed test, if it is detected that the test has failed and the defect candidates have already been set; an output unit that outputs, if there is one functional unit that is a defect candidate, a message indicating that the one functional unit has a defect; a selection unit that selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, if there are a plurality of functional units that are defect candidates, such that a functional unit used in any test determined by the test sequence to be executed next narrows down the plurality of defect candidate functional units; A program characterized by functioning as

2. a candidate set representing the defect candidates is stored in a storage area of ​​the computer; If it is detected that the test has failed and the candidate set is an empty set, the setting unit adds a functional unit used in the failed test to the candidate set; If it is detected that the test has failed and the candidate set is not an empty set, the narrowing unit removes from the candidate set functional units that are not used in the failed test; If the candidate set has one element, the output unit outputs a message indicating that the functional unit that is the one element has a defect.

2. The program according to claim 1,

3. When all of the plurality of test sequences are executed and there are a plurality of functional units that are defect candidates, the output unit outputs a message indicating that at least one of the plurality of defect candidate functional units has a defect.

2. The program according to claim 1,

4. If the total time required to execute the executed test sequences among the plurality of test sequences exceeds a predetermined time and there are a plurality of functional units that are defect candidates, the output unit outputs a message indicating that at least one of the plurality of defect candidate functional units has a defect.

2. The program according to claim 1,

5. If all of the plurality of test sequences are executed without detecting any failure, the output unit outputs a message indicating that there is no defect in any of the one or more functional units.

2. The program according to claim 1,

6. Among the unexecuted test sequences, a test sequence that defines a test that utilizes a functional unit that narrows down the plurality of functional units that are defect candidates and that has the shortest expected time from the start of execution of the test sequence to the end of the test is selected as the test sequence to be executed next.

2. The program according to claim 1,

7. Among the unexecuted test sequences, a test sequence that defines a test that uses functional units that result in the smallest number of functional units after narrowing down the plurality of functional units that are defect candidates is selected as the test sequence to be executed next.

2. The program according to claim 1,

8. From a plurality of combinations of a failure set whose elements are tests detected as having failed by the detection unit, A combination in which one functional unit is used in all tests included in the combination An extractor that attempts to extract Furthermore, If the extraction is successful, the output unit outputs a message indicating that one functional unit related to the extracted combination has a defect.

2. The program according to claim 1,

9. A control device for controlling an order in which a plurality of test sequences are executed, each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; The control device a detection unit for detecting a failure of a test during execution of a test sequence; a setting unit that sets a functional unit used in the failed test as a defect candidate if the test has failed and a defect candidate has not yet been set; a narrowing-down unit that narrows down the functional units that are the defect candidates to the functional units used in the failed test, if it is detected that the test has failed and the defect candidates have already been set; an output unit that outputs, if there is one functional unit that is a defect candidate, a message indicating that the one functional unit has a defect; a selection unit that selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, if the plurality of functional units are defect candidates, such that the plurality of functional units that are defect candidates are narrowed down to functional units to be used in a test determined by the test sequence to be executed next; A control device comprising:

10. A control method executed by a control device that controls an order in which a plurality of test sequences are executed, comprising: each test sequence of the plurality of test sequences defines an order for executing one or more tests; one or more functional units are utilized in performing each test of the one or more tests; a detecting step in which the control device detects that a test has failed during execution of a test sequence; a setting step in which, when it is detected that the test has failed and a failure candidate has not yet been set, the control device sets the functional unit used in the failed test as the failure candidate; a narrowing-down process in which, if it is detected that the test has failed and the defect candidates have already been set, the control device narrows down the defect candidate functional units to the functional units used in the failed test; an output step in which, if there is one functional unit that is a defect candidate, the control device outputs an output indicating that the one functional unit has a defect; a selection step in which, if there are a plurality of functional units that are defect candidates, the control device selects a test sequence to be executed next from among the plurality of test sequences that have not yet been executed, the selection step being performed by the selection unit, such that a functional unit used in any test determined by the test sequence to be executed next narrows down the plurality of defect candidate functional units; A control method comprising:

Images