Test execution device, test execution method, and program
The test execution device addresses the challenge of completing multiple tests within a limited time by recalculating the execution order based on the degree of similarity with a failed test, enhancing the detection of undiscovered defects.
Patent Information
- Application Number
- JP2023191122
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-11-08
- Publication Date
- 2025-05-20
- Estimated Expiration
- 2043-11-08
AI Technical Summary
When multiple tests are executed in a set order and a specific test fails, processing delays may prevent the completion of remaining tests within a limited time, leading to a low likelihood of discovering undiscovered defects.
A test execution device that detects failures and recalculates the execution order of remaining tests based on the degree of similarity with the failed test, prioritizing tests with lower degrees of failure.
This approach allows for the completion of as many tests as possible within the limited time and increases the likelihood of discovering undiscovered defects by adjusting the execution order based on the degree of failure.
Smart Images
Figure 2025078504000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a test execution device, a test execution method, and a program. [Background technology]
[0002] Various techniques for executing tests to check the functions and operations of a program are known. For example, Patent Document 1 discloses a technique in which, when an error occurs when a specific test is executed, other tests in which a similar error occurs are associated with the specific test as tests having a co-occurrence relationship with the error, and when an error occurs when a test is executed, the tests having the co-occurrence relationship are added and executed. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 621721 Summary of the Invention [Problem to be solved by the invention]
[0004] On the other hand, when multiple tests are executed in a set order and all tests must be completed within a limited time, if a specific test fails during execution, processing delays may prevent the remaining tests from being completed within the limited time. In such a case, even if a test similar to the failed test among the remaining tests is executed, the test may fail due to a defect similar to that of the failed test, and the possibility of discovering undiscovered defects is low. Therefore, there is a demand for adjusting the execution order of the remaining tests based on the degree of failure in order to complete execution of as many tests as possible within a limited time and discover undiscovered defects.
[0005] The present invention is devised to solve the above-mentioned problems, and aims to provide a test execution device, test execution method, and program that, when a failure is detected when multiple tests are executed in a set order, can re-set the execution order of tests that have not yet been executed based on the degree of failure. [Means for solving the problem]
[0006] A test execution apparatus according to a first aspect of the present invention comprises: A test execution device for executing a plurality of tests in a set sequence, comprising: A detection unit that detects a failure of any one of the plurality of tests; a calculation unit that, when a failure is detected by the detection unit, calculates a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculates a degree of failure of each of the tests that have not yet been executed based on the similarity; and a resetting unit that resets an order in which the tests that have not yet been executed are executed based on the degree of failure calculated by the calculating unit; and The present invention is characterized by comprising:
[0007] In addition, in the test execution device according to the above aspect, The resetting unit resets the order in which the tests that have not yet been executed are executed in order from the lowest degree of failure. It is characterized by:
[0008] In addition, in the test execution device according to the above aspect, The resetting unit resets the order in which the tests that have not yet been executed are executed in order of the degree of failure being closest to a representative value in a distribution of the degree of failure. It is characterized by:
[0009] In addition, in the test execution device according to the above aspect, The calculation unit calculates, as the similarity, a similarity between the test in which the failure was most recently detected and each of the tests that have not yet been executed. It is characterized by:
[0010] In addition, in the test execution device according to the above aspect, The calculation unit calculates, as the similarity, a similarity between one or more tests in which a failure has been detected from the start of execution of the plurality of tests to the present time and each of the tests that have not yet been executed. It is characterized by:
[0011] In addition, in the test execution device according to the above aspect, The calculation unit calculates the similarity based on a function, a module, or a library used by the test target. It is characterized by:
[0012] In addition, in the test execution device according to the above aspect, The calculation unit calculates the similarity based on elements appearing in a text associated with the test subject. It is characterized by:
[0013] In addition, in the test execution device according to the above aspect, The calculation unit calculates the similarity based on a predetermined group of the tests. It is characterized by:
[0014] In addition, in the test execution device according to the above aspect, The calculation unit calculates the similarity based on whether results of past executions of the test co-occur. It is characterized by:
[0015] In addition, in the test execution device according to the above aspect, The calculation unit calculates the similarity as the degree of failure. It is characterized by:
[0016] In addition, in the test execution device according to the above aspect, When a plurality of similarities are obtained for one of the tests that have not yet been executed, the calculation unit calculates a representative value of the plurality of similarities as the degree of failure. It is characterized by:
[0017] In addition, in the test execution device according to the above aspect, When a plurality of similarities are obtained for one of the tests that have not yet been executed, the calculation unit calculates a sum of the plurality of similarities obtained as the degree of failure. It is characterized by:
[0018] A test execution method according to a second aspect of the present invention comprises: 1. A test execution method executed by a test execution device that executes a plurality of tests in a set order, comprising: a detection step of detecting a failure of any one of the plurality of tests; a calculation step of calculating, when a failure is detected in the detection step, a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculating a degree of failure of each of the tests that have not yet been executed based on the similarity; a resetting step of resetting an order in which the tests that have not yet been executed are executed based on the degree of failure calculated in the calculation step; The present invention is characterized by comprising:
[0019] A program according to a third aspect of the present invention comprises: A program that causes a computer to function as a test execution device that executes a plurality of tests in a set sequence, comprising: The computer, a detection unit that detects a failure of any one of the plurality of tests; a calculation unit that, when a failure is detected by the detection unit, calculates a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculates a degree of failure of each of the tests that have not yet been executed based on the similarity; a resetting unit that resets the order in which the tests that have not yet been executed are executed based on the degree of failure calculated by the calculating unit; The present invention is characterized in that it functions as a
[0020] The above program may be recorded on a non-transitory recording medium. The non-transitory recording medium can be distributed and sold independently of the computer. Here, the non-transitory recording medium refers to a tangible recording medium. Examples of the non-transitory recording medium include a compact disc, a flexible disc, a hard disk, a magneto-optical disk, a digital video disk, a magnetic tape, and a semiconductor memory. In addition, the transitory recording medium refers to the transmission medium (propagation signal) itself. Examples of the transitory recording medium include an electric signal, an optical signal, and an electromagnetic wave. In addition, the temporary storage area is an area for temporarily storing data and programs, and is, for example, a volatile memory such as a RAM (Random Access Memory). Effect of the Invention
[0021] According to the present invention, it is possible to provide a test execution device, a test execution method, and a program that, when a failure is detected when multiple tests are executed in a set order, can re-set the execution order of tests that have not yet been executed based on the degree of failure. [Brief description of the drawings]
[0022] [Figure 1] FIG. 2 is a diagram illustrating a relationship between a test execution device and a server according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a hardware configuration of a test execution apparatus according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a functional configuration of a test execution device according to an embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of test information according to the embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of execution order information according to the embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of execution result information according to the embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of execution order information according to the embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a distribution of failure degrees according to the embodiment; [Figure 9] FIG. 11 is a diagram illustrating an example of execution order information according to the embodiment. [Figure 10] 1 is a flowchart illustrating a test execution process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0023] (1. Overall Configuration) As shown in FIG. 1, a test execution apparatus 100 according to an embodiment of the present invention is communicatively connected to a server 200 via a computer communication network 300 such as the Internet.
[0024] The test execution device 100 is a device that executes a plurality of tests in a set sequence. For example, the test execution device 100 executes a task of executing 1000 tests in a set sequence once a day, and repeats the execution of the task every day. The tests are intended to check the operation of the server 200, for example.
[0025] The server 200 provides various services, such as betting on horse races and viewing race footage.
[0026] (2. Hardware configuration of the test execution device) FIG. 2 is a block diagram showing the hardware configuration of the test execution device 100. As shown in FIG.
[0027] 2, the test execution apparatus 100 comprises a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a recording medium 14, an output device 15, a communication device 16, and an operation device 17. The components are connected by a bus 18.
[0028] The CPU 11 controls the overall operation of the test execution apparatus 100, and is connected to each of the components to exchange control signals and data.
[0029] The ROM 12 stores operating programs and various data necessary for controlling the overall operation of the test execution device 100.
[0030] The RAM 13 is for temporarily storing data and programs, and holds programs and data read from the recording medium 14, as well as other data necessary for communication.
[0031] The recording medium 14 is composed of a hard disk, a flash memory, or the like, and records data to be processed by the test execution device 100 .
[0032] The output device 15 includes a display device such as an LCD (Liquid Crystal Display) and an audio output device such as a speaker. The output device 15 outputs, for example, data output from the CPU 11 under the control of the CPU 11.
[0033] The communication device 16 includes a communication interface for connecting the test execution apparatus 100 to a computer communication network such as the Internet, and communicates with other information processing apparatuses via the communication device 16 .
[0034] The operation device 17 includes input devices such as buttons, a keyboard, a touch panel, a microphone, etc. The operation device 17 receives an operation input from a user of the test execution apparatus 100, and outputs a signal corresponding to the received operation input to the CPU 11.
[0035] (3. Functional configuration of the test execution device according to the embodiment) 3, the test execution apparatus 100 functionally comprises a detection unit 101, a calculation unit 102, and a resetting unit 103. In this embodiment, the CPU 11 and the communication device 16 cooperate to function as the detection unit 101, and the CPU 11 functions as the calculation unit 102 and the resetting unit 103.
[0036] Furthermore, the test execution device 100 stores, for each of the multiple tests, test information indicating the content of the test, execution order information indicating the order in which the tests were executed, and execution result information indicating the results of the test execution in the recording medium 14.
[0037] In the following, the functions of each part and the information stored therein will be explained using an example in which the test execution device 100 executes a task including 1,000 tests (test IDs "T1" to "T1000") to check the operation of a service provided by the server 200.
[0038] An example of test information is shown in Fig. 4. The test information includes a test ID for identifying a test, the name of the program to be tested, functions, modules, and libraries used by the program, an ID for identifying the source code of the program, and a predetermined test group, all of which are associated with each other.
[0039] A predetermined group of tests is, for example, a group arbitrarily determined by a user of the test execution device 100 based on the commonality of functions provided by the programs to be tested. Regarding the tests in Fig. 4, the tests are divided into "Group 1" consisting of tests with test ID "1" to test ID "200", "Group 2" consisting of tests with test ID "201" to test ID "T400", "Group 3" consisting of tests with test ID "401" to test ID "600", "Group 4" consisting of tests with test ID "601" to test ID "800", and "Group 5" consisting of tests with test ID "801" to test ID "T1000".
[0040] For example, the record in the first row of Figure 4 indicates that the test with test ID "T1" is a test to verify the operation of a program named "A0-B0-C0", that the program uses the functions "Function 1" and "Function 2", that the program uses the modules "Module 1", "Module 2", and "Module 3", and that the program uses "Library 1", that the source code of the program is source code identified by source code ID "S1", and that the test with test ID "T1" belongs to "Group 1".
[0041] An example of the execution order information is shown in Fig. 5. The execution order information includes a test ID for identifying a test and an execution order of the test, which are associated with each other. It is assumed that the execution order information in Fig. 5 is initially set.
[0042] For example, the record in the first row of Fig. 5 indicates that the test with test ID "T1" will be executed "1st." The test execution device 100 executes the tests from test ID "T1" to test ID "T1000" in the execution order shown in Fig. 5.
[0043] FIG. 6 shows an example of the execution result information. The execution result information includes a test ID for identifying a test and a test success / failure indicating whether the test was successful or failed, in association with each other. The success / failure is recorded for each date and time when the task was executed. "Date and time k" (k is a natural number of 2 or more) indicates the current time when the test execution device 100 has executed 1000 test tasks, and "Date and time 1" to "Date and time (k-1)" indicate the past times when the test execution device 100 executed 1000 test tasks. The execution result information in FIG. 6 indicates that at "Date and time k", execution of tests up to test ID "T500" has been completed. Note that the execution result information is registered automatically by the test program or manually by the tester when a test is executed.
[0044] For example, the record in the first row of FIG. 6 indicates that the test with test ID "T1" was "successful" at "date and time 1", "successful" at "date and time (k-1)", and "successful" at "date and time k".
[0045] The detection unit 101 in FIG. 3 detects a failure in any one of a plurality of tests.
[0046] For example, if the test execution device 100 executes tests in the execution order shown in the execution order information in Figure 5, starting with test ID "T1", and the test with test ID "T500" in the execution order "500" fails, the detection unit 101 will detect the failure of the test with test ID "T500".
[0047] When a failure is detected by the detection unit 101, the calculation unit 102 in Fig. 3 calculates the similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the multiple tests, and calculates the degree of failure of each of the tests that have not yet been executed based on the similarity. The degree of failure indicates the possibility that a test that has not yet been executed will fail if it is executed. Therefore, the calculation unit 102 calculates the possibility that each of the tests that have not yet been executed will fail based on the similarity between the test in which the failure was detected and each of the tests that have not yet been executed.
[0048] For example, if the detection unit 101 detects a failure of a test with test ID "T500", the tests that have not yet been executed among the multiple tests are tests with test IDs "T501" to "T1000". The calculation unit 102 calculates the degree of failure for each of the tests with test IDs "T501" to "T1000" based on the similarity with the test in which the failure was detected.
[0049] Here, the tests in which failures have been detected and for which similarity is to be calculated include tests in which failures have been detected recently or in the past.
[0050] Specifically, the calculation unit 102 calculates the similarity between the test in which a failure was most recently detected and each of the tests that have not yet been executed, and calculates the similarity between the test in which a failure was detected and each of the tests among the multiple tests that have not yet been executed.
[0051] For example, when the detection unit 101 detects a failure of a test with test ID "T500", the test whose failure was detected most recently is the test with test ID "T500". Therefore, the calculation unit 102 calculates the similarity between each of the tests with test IDs "T501" to "T1000" and test ID "T500". In this case, one similarity is obtained for each of the tests with test IDs "T501" to "T1000".
[0052] Alternatively, the calculation unit 102 calculates the similarity between one or more tests in which a failure has been detected since the start of execution of the multiple tests up to the present and a test that has not yet been executed, as the similarity between the test in which a failure has been detected and each of the multiple tests that has not yet been executed.
[0053] For example, as shown in the execution result information of FIG. 6, before a failure is detected in the test with test ID "T500", failures are also detected in the tests with test ID "T100" and test ID "T400". In this case, the one or more tests in which failures have been detected from the start of execution of the multiple tests to the present are the tests with test ID "T100", test ID "T400", and test ID "T500". Therefore, the calculation unit 102 calculates the similarity between each of the tests with test ID "T501" to test ID "T1000" and each of test ID "T100", test ID "T400", and test ID "T500". In this case, three similarities are obtained for each of the tests with test ID "T501" to test ID "T1000".
[0054] The similarity is determined from various perspectives.
[0055] Specifically, the calculation unit 102 calculates the similarity based on the functions, modules, or libraries used by the test target.
[0056] 4, compares the functions, modules, or libraries of each program subject to tests with test IDs "T501" to "T1000" with the functions, modules, or libraries of the program subject to test ID "T500", and calculates a similarity score based on the number of common functions, modules, or libraries used. For example, the program subject to test ID "T501" (name "A5-B0-C0") has one function and two libraries in common with the program subject to test ID "T500" (name "A4-B9-C9"), so the calculation unit 102 calculates a similarity score of "3".
[0057] Therefore, the more functions, modules, or libraries that a test subject that has not yet been executed uses in common with a test subject in which a failure has been detected, the higher the required similarity is.
[0058] The calculation unit 102 also calculates the degree of similarity based on the degree to which elements appearing in the text associated with the test subject are common or different.
[0059] The text associated with the test target is, for example, the name of the target, a specification, a manual, or source code. Moreover, the elements appearing in the text are, for example, words or identifiers. The degree to which elements are common is, for example, the number of common elements contained in two texts. Moreover, the degree to which elements are different is, for example, the number of different elements contained in two texts.
[0060] For example, the calculation unit 102 refers to the test information in FIG. 4, compares the names of the programs to be tested from test ID "T501" to test ID "T1000" with the name "A4-B9-C9" of the program to be tested for test ID "T500", and calculates a similarity score based on the number of common words or identifiers. For example, the name "A5-B0-C0" of the program to be tested for test ID "T501" has 0 common identifiers with the name "A4-B9-C9" of the program to be tested for test ID "T500", so the calculation unit 102 calculates a similarity score of "0". On the other hand, the name "A9-B9-C9" of the program to be tested for test ID "T1000" has two common identifiers with the name "A4-B9-C9" of the program to be tested for test ID "T500", so the calculation unit 102 calculates a similarity score of "2".
[0061] Also, for example, the calculation unit 102 refers to the test information in FIG. 4 to identify the source code of each program subject to the test with test ID "T501" to test ID "T1000" and the source code of the program subject to the test with test ID "T500", and compares the identified source codes. Then, the calculation unit 102 calculates a similarity score based on the number of common words or identifiers. For example, if the source code with source code ID "S501" of the program subject to the test with test ID "T501" contains 10 words and identifiers that are included in the source code with source code ID "S501" of the program subject to the test with test ID "T500", the calculation unit 102 calculates the similarity score as "10".
[0062] Therefore, the more elements that a text associated with a test target that has not yet been executed has in common with a text associated with a test target for which a failure has been detected, the higher the degree of similarity that is determined to be, or the more elements that a text associated with a test target that has not yet been executed has that differ from a text associated with a test target for which a failure has been detected, the lower the degree of similarity that is determined to be.
[0063] The calculation unit 102 may also determine the similarity based on a group of predefined tests.
[0064] 4, the calculation unit 102 calculates a similarity score based on whether the group of each test from test ID "T501" to test ID "T1000" is the same as "group 3" of the test with test ID "T500". For example, since the test with test ID "T501" belongs to "group 3" like the test with test ID "T500", the calculation unit 102 calculates the similarity score as "1". On the other hand, since the test with test ID "T601" belongs to "group 4" different from "group 3" of the test with test ID "T500", the calculation unit 102 calculates the similarity score as "0".
[0065] Therefore, if a group to which a test that has not yet been executed belongs is the same group as a test in which a failure has been detected, the similarity is determined to be high.
[0066] The calculation unit 102 also calculates the similarity based on whether or not results of tests previously performed co-occur.
[0067] A co-occurring outcome means that when one test is run in conjunction with another test, both tests fail or both tests succeed.
[0068] 6, the calculation unit 102 references the success / failure of the test at "date / time (k-1)" that was executed immediately before "date / time k", and calculates the similarity score based on whether the success / failure of tests from test ID "T501" to test ID "T1000" is the same as the success / failure of the test at test ID "T500". For example, the success / failure of the test at "date / time (k-1)" with test ID "T501" is "fail", and the success / failure of the test at "date / time (k-1)" with test ID "T500" is "fail", so the success / failures are the same, and therefore the calculation unit 102 calculates the similarity score as "1". On the other hand, the success / failure of the test with test ID "T601" at "date / time (k-1)" is "success", which is different from the success / failure of the test with test ID "T500" at "date / time (k-1)", so the calculation unit 102 calculates the similarity score as "0". Note that the past success / failure is not limited to the success / failure of the most recently executed test, but may be, for example, the success / failure of all tests executed in the past. For example, the calculation unit 102 calculates the similarity for each of the tests with test ID "T501" to test ID "T1000" based on whether or not the success / failure is the same as the success / failure of the test with test ID "T500" at "date / time 1" to "date / time (k-1)". In this case, (k-1) similarities are obtained for each of the tests with test ID "T501" to test ID "T1000".
[0069] Therefore, the more the results of a test that has not yet been executed and a test whose failure has been detected occur together, the higher the degree of similarity that is obtained.
[0070] Next, the calculation unit 102 calculates the similarity as the degree of failure.
[0071] For example, if a similarity of "1" is obtained for a test with test ID "T501", the calculation unit 102 calculates the degree of failure of the test with test ID "T501" as "1".
[0072] Furthermore, when multiple similarities are obtained for one of the tests that has not yet been executed, the calculation unit 102 calculates a representative value of the multiple obtained similarities as the degree of failure.
[0073] The representative value of the multiple similarities obtained is, for example, the average value, median, or mode of the multiple similarities obtained.
[0074] For example, if the calculation unit 102 obtains similarities of "1", "0", and "1" for the test with test ID "T501" with test ID "T100", test ID "T400", and test ID "T500", respectively, the calculation unit 102 calculates the average value of these similarities to be "0.67" as the degree to which the test with test ID "T501" will fail.
[0075] Alternatively, when multiple similarities are obtained for one of the tests that has not yet been executed, the calculation unit 102 calculates the sum of the multiple obtained similarities as the degree of failure.
[0076] For example, if the calculation unit 102 obtains similarities of "1", "0", and "1" between the test with test ID "T501" and test ID "T100", test ID "T400", and test ID "T500", the calculation unit 102 calculates the sum of these similarities to be "2" as the degree to which the test with test ID "T501" will fail.
[0077] Therefore, the degree of failure is determined to be higher as the degree of similarity obtained from various viewpoints is higher. For example, the more functions, modules, or libraries that the target of the test that has not yet been executed uses in common with the target of the test whose failure was detected, the higher the degree of failure of the test that has not yet been executed. Also, the more elements that the text associated with the target of the test that has not yet been executed contains in common with the text associated with the target of the test whose failure was detected, the higher the degree of failure of the test that has not yet been executed. Also, if the group to which the test that has not yet been executed belongs belongs to the same group as the test whose failure was detected, the higher the degree of failure of the test that has not yet been executed. Also, the more the results of the test that has not yet been executed and the test whose failure was detected co-occur, the higher the degree of failure of the test that has not yet been executed.
[0078] When the degree of failure is calculated by the calculation unit 102, the resetting unit 103 in FIG. 3 resets the order in which tests that have not yet been executed are to be executed, based on the degree of failure.
[0079] Specifically, the resetting unit 103 resets the order in which tests that have not yet been executed are executed in ascending order of the degree of failure.
[0080] For example, if the degree of failure of tests from test ID "T501" to test ID "T1000" decreases in the order of test IDs "T1000", "T999", "T998", "T997", ..., "T504", "T503", "T502", and "T501", the resetting unit 103 resets the execution order of tests from test ID "T501" to test ID "T1000" as shown in the execution order information of Figure 7.
[0081] When the test execution order is reset, the test execution device 100 refers to the execution order information in Fig. 7 and starts execution from the test with test ID "T1000" that is execution order number "501". Therefore, the test execution device 100 postpones tests that are likely to fail and prioritizes tests that are unlikely to fail.
[0082] Alternatively, the resetting unit 103 resets the order in which tests that have not yet been executed are executed in order of the degree of failure that is closest to a representative value in the distribution of the degrees of failure.
[0083] The representative value in the distribution of the degree of failure is, for example, the mean value, the median value, or the mode value in the distribution.
[0084] An example of the distribution of the degree of failure is shown in Fig. 8. For example, for tests from test ID "T501" to test ID "T1000", if the degree of failure is closer to the representative value XR of the distribution in Fig. 8 in the order of test IDs "T750", "T751", "T749", "T752", ..., "T502", "T999", "T501", and "T1000", the resetting unit 103 resets the execution order of tests from test ID "T501" to test ID "T1000" as shown in the execution order information in Fig. 9.
[0085] When the test execution order is reset, the test execution device 100 refers to the execution order information in Fig. 9 and starts execution from test with test ID "T750" in the execution order No. "501". Therefore, the test execution device 100 postpones tests that are likely to fail and tests that are unlikely to fail, i.e., tests that are likely to succeed, and executes the remaining tests.
[0086] In addition, in the case where a plurality of tests are executed periodically, if there are unexecuted tests remaining at the time when the next task is started, the execution of the remaining tests is abandoned.
[0087] For example, suppose that the tasks for executing tests with test IDs "T1" to "T1000" were executed at date and time k in the execution order shown in FIG. 7, but the tests with test IDs "T501" to "T600" were not completed within the expected time for task completion. In this case, the test execution device 100 does not execute the tests with test IDs "T501" to "T600", and starts the tasks for executing tests with test IDs "T1" to "T1000" at the next predetermined date and time (k+1). Note that if the task executed at the previous date and time k was not completed within the expected time, the test execution device 100 may execute the tests with test IDs "T1" to "T1000" at date and time (k+1) in the reset execution order shown in FIG. 7, or in the initially set execution order shown in FIG. 5.
[0088] (4. Operation of the Test Execution Apparatus of the Embodiment) The operation of the test execution device 100 of this embodiment will be described with reference to Fig. 10. For example, when the test execution device 100 receives an instruction to start a task for executing multiple tests, it starts the test execution process shown in Fig. 10. Note that when the test execution process in Fig. 10 starts, i is 1, and the total number of tests is N.
[0089] The test execution device 100 executes a test in execution order i (step S101).
[0090] For example, the test execution device 100 executes a test in execution order i according to the initially set execution order information in FIG.
[0091] The detection unit 101 judges whether or not a failure of a test being executed has been detected (step S102). If the detection unit 101 judges that a failure of a test being executed has been detected (step S102; YES), the calculation unit 102 calculates the similarity between the test in which the failure has been detected and each of the tests that have not yet been executed among the multiple tests (step S103). On the other hand, if the detection unit 101 judges that a failure of a test being executed has not been detected (step S102; NO), the calculation unit 102 proceeds to step S106.
[0092] For example, if the detection unit 101 determines that it has detected a failure of test with test ID "T500" and execution order number "500", it calculates the similarity between the test with test ID "T500" and each of the tests with test IDs "T501" to "T1000" that have not yet been executed. On the other hand, if the detection unit 101 determines that it has not detected a failure of test with test ID "T500" and execution order number "500", it proceeds to step S106.
[0093] The calculation unit 102 calculates the degree of failure of each of the tests that have not yet been executed based on the similarity (step S104).
[0094] For example, if a similarity of "1" is obtained for a test with test ID "T501", the calculation unit 102 calculates the degree of failure of the test with test ID "T501" as "1".
[0095] The resetting unit 103 resets the execution order of the tests that have not yet been executed, that is, the tests from execution order (i+1) to execution order N, based on the degree of failure (step S105).
[0096] For example, if the degree of failure of tests from test ID "T501" to test ID "T1000" decreases in the order of test IDs "T1000", "T999", "T998", "T997", ..., "T504", "T503", "T502", and "T501", the resetting unit 103 resets the execution order of tests from test ID "T501" to test ID "T1000" as shown in the execution order information of Figure 7.
[0097] The test execution device 100 increments the execution order i (step S106). Then, the test execution device 100 judges whether the execution of the test of execution order i will be completed within the expected time to complete the task (step S107). If the test execution device 100 judges that the execution of the test of execution order i will be completed within the expected time to complete the task (step S107; YES), the test execution device 100 returns to step S101. On the other hand, if the test execution device 100 judges that the execution of the test of execution order i will not be completed within the expected time to complete the task (step S107; NO), the process of FIG. 10 ends.
[0098] For example, the test execution device 100 increments the execution order i (=500), and if it determines that test "501" (test ID "T1000") in the execution order will be completed within the expected time of the task (e.g., within 24 hours from the start of the task), it executes test "501" (test ID "T1000") in the execution order. On the other hand, if the test execution device 100 determines that test "501" (test ID "T1000") in the execution order will not be completed within the expected time of the task, it ends the test execution process in FIG. 10 without executing tests with test IDs "T501" to "T1000" in the execution order that have not yet been executed.
[0099] According to this embodiment, when multiple tests are executed in a set order, if a failure of a test is detected, the execution order of the remaining tests that have not yet been executed can be reset based on the degree of failure calculated from the similarity with the failed test. In this way, when multiple tests must be executed within a limited time, the execution order of the tests can be adjusted based on whether or not they are similar to the failed test, in order to complete execution of as many tests as possible and discover undiscovered defects.
[0100] Furthermore, according to this embodiment, the execution order of tests that have not yet been executed can be reset in order of decreasing degree of failure based on the similarity with the failed test. As a result, when multiple tests must be executed within a limited time, tests that are more likely to fail can be postponed, and as many tests as possible can be executed within the limited time. This increases the possibility of completing the execution of all multiple tests. Furthermore, since tests that have different characteristics from the failed test can be executed with priority, undiscovered defects can be found as early as possible.
[0101] Furthermore, according to this embodiment, the execution order of tests that have not yet been executed can be reset in order of the degree of failure that is determined based on the similarity with the failed test to a representative value. This allows tests that are likely to fail and tests that are likely to succeed to be executed later, and the remaining tests to be executed later. Therefore, tests that have different characteristics from the failed test and are likely to fail are executed first, so that undiscovered defects can be found earlier.
[0102] Furthermore, according to this embodiment, the similarity used to calculate the degree of failure can be calculated based on whether the test subjects use common functions, modules, or libraries, whether the text elements associated with the subjects are common or different, whether they belong to a common group, and whether the results of past tests co-occur. This makes it possible to calculate the similarity based on various characteristics of the tests, and to accurately calculate the similarity between a failed test and a test that has not yet been executed.
[0103] Furthermore, according to the present embodiment, the similarity used to calculate the degree of failure is calculated for the most recent or past failed tests, which allows the similarity to be calculated taking into account the tendency of the most recent or past failed tests.
[0104] Furthermore, according to this embodiment, the degree of failure is calculated based on one or more similarities, so that the degree of failure of a test that has not yet been executed can be calculated based on the similarity to a specific failed test or based on the similarity to multiple failed tests.
[0105] (5. Modifications) Although the embodiment of the present invention has been described above, various modifications and applications are possible in carrying out the present invention.
[0106] In the above embodiment, the similarity is calculated based on a plurality of viewpoints, but may be calculated by combining a plurality of viewpoints. For example, the sum of the similarity calculated based on the function, module, or library used by the test subject and the similarity calculated based on the degree of commonality or difference between elements appearing in the text associated with the test subject may be used as the similarity used to calculate the degree of failure.
[0107] In addition, in the above embodiment, an example of the method of calculating the similarity is shown, but the present invention is not limited to this. For example, the calculation unit 102 may calculate the similarity by using a trained model that inputs test information of a test in which a failure has been detected and a test that has not yet been executed and outputs a similarity. For example, the trained model is generated by training a neural network or the like using the test information of a test in which a failure has been detected and a test that has not yet been executed, and the similarity between a test in which a failure has been detected and a test that has not yet been executed as training data. Machine learning, deep learning, and the like can be applied to the learning, and it is also possible to adopt a so-called artificial intelligence service. The trained model may be generated by the test execution device 100 or another device.
[0108] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]
[0109] According to the present invention, it is possible to provide a test execution device, a test execution method, and a program that, when a failure is detected when multiple tests are executed in a set order, can re-set the execution order of tests that have not yet been executed based on the degree of failure. [Explanation of symbols]
[0110] 11 CPU 12 ROM 13 RAM 14 Recording media 15 Output Devices 16 Communication Devices 17 Operating Devices 18 Bus 100 Test Execution Device 101 Detection unit 102 Calculation Department 103 Resetting section 200 Servers 300 Computer Communication Network
Claims
1. A test execution device for executing a plurality of tests in a set sequence, comprising: A detection unit that detects a failure of any one of the plurality of tests; a calculation unit that, when a failure is detected by the detection unit, calculates a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculates a degree of failure of each of the tests that have not yet been executed based on the similarity; and a resetting unit that resets an order in which the tests that have not yet been executed are executed based on the degree of failure calculated by the calculating unit; and A test execution device comprising:
2. The resetting unit resets the order in which the tests that have not yet been executed are executed in order from the lowest degree of failure.
2. The test execution apparatus according to claim 1.
3. The resetting unit resets the order in which the tests that have not yet been executed are executed in order of the degree of failure being closest to a representative value in a distribution of the degree of failure.
2. The test execution apparatus according to claim 1.
4. The calculation unit calculates, as the similarity, a similarity between the test in which the failure was most recently detected and each of the tests that have not yet been executed.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
5. The calculation unit calculates, as the similarity, a similarity between one or more tests in which a failure has been detected from the start of execution of the plurality of tests to the present time and each of the tests that have not yet been executed.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
6. The calculation unit calculates the similarity based on a function, a module, or a library used by the test target.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
7. The calculation unit calculates the similarity based on elements appearing in a text associated with the test subject.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
8. The calculation unit calculates the similarity based on a predetermined group of the tests.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
9. The calculation unit calculates the similarity based on whether results of past executions of the test co-occur.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
10. The calculation unit calculates the similarity as the degree of failure.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
11. When a plurality of similarities are obtained for one of the tests that have not yet been executed, the calculation unit calculates a representative value of the plurality of similarities as the degree of failure.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
12. When a plurality of similarities are obtained for one of the tests that has not yet been executed, the calculation unit calculates a sum of the plurality of similarities obtained as the degree of failure.
4. A test execution apparatus according to claim 1, wherein the test execution apparatus executes a test execution command.
13. 1. A test execution method executed by a test execution device that executes a plurality of tests in a set order, comprising: a detection step of detecting a failure of any one of the plurality of tests; a calculation step of calculating, when a failure is detected in the detection step, a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculating a degree of failure of each of the tests that have not yet been executed based on the similarity; a resetting step of resetting an order in which the tests that have not yet been executed are executed based on the degree of failure calculated in the calculation step; 13. A method of executing a test comprising:
14. A program that causes a computer to function as a test execution device that executes a plurality of tests in a set sequence, comprising: The computer, a detection unit that detects a failure of any one of the plurality of tests; a calculation unit that, when a failure is detected by the detection unit, calculates a similarity between the test in which the failure was detected and each of the tests that have not yet been executed among the plurality of tests, and calculates a degree of failure of each of the tests that have not yet been executed based on the similarity; a resetting unit that resets the order in which the tests that have not yet been executed are executed based on the degree of failure calculated by the calculating unit; A program characterized by causing the program to function as a
Citation Information
Patent Citations
Program, information processor, and testing method
JP2014049066A
Epoxy resin composition
JP1987001721A