A calculation device, calculation method, and program for determining the priority of test cases.

Abstract

This invention provides a calculation device, calculation method, and program that enable more appropriate prioritization of test cases. [Solution] The calculation device 100 includes: a similarity calculation unit 140 that calculates the similarity between the definition information of each test case and the definition information of other test cases related to other projects for each of the multiple test cases related to the selected project; an estimation unit 150 that estimates the probability that a new execution of each test case will fail based on the execution history of each test case related to the selected project, the execution history of other test cases and the calculated similarity; and a priority determination unit 160 that determines the priority of each test case based on the estimated probability.

Description

[Technical Field]

[0001] This invention relates to a calculation device, calculation method, and program for calculating the priority of test cases. [Background technology]

[0002] When making changes to a program, such as adding features or fixing bugs, it is common practice to conduct tests to ensure that no new problems arise as a result of those changes. Because there are limitations to the effort and time available for testing, it is common practice to prioritize and select test cases that should be executed first.

[0003] For example, Patent Document 1 discloses a test case extraction device that extracts test cases with high priority according to the number of past failures from a group of pre-created test cases, which are multiple test cases that have been created in advance and repeatedly tested. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2007-102475 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, test cases that, for example, manually verify that the UI (User Interface) functions correctly are often performed only once or less due to constraints on time, cost, and human resources. Therefore, when there are many test cases that are performed manually, the test case extraction device in Patent Document 1 has room for improvement in terms of appropriately determining the priority of test cases, such as assigning the same priority to a large number of test cases.

[0006] This invention has been made in view of the above circumstances, and aims to provide a calculation device, calculation method, and program that can more appropriately determine the priority of test cases. [Means for solving the problem]

[0007] To solve the above problems, the calculation device according to the present invention is A similarity calculation unit that determines the similarity between the definition information of each test case of a set of test cases related to a selected project and the definition information of other test cases related to other projects, An estimation unit estimates the probability that a new execution of each test case will fail, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the similarity calculated by the similarity calculation unit. A priority determination unit determines the priority of each of the test cases based on the probability estimated by the estimation unit, It is equipped with. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a calculation device, a calculation method, and a program that can more appropriately determine the priority of test cases. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram showing the coordination between the calculation device and other equipment. [Figure 2] This is an explanatory diagram showing the functional configuration of the calculation device. [Figure 3] Figure 2 shows an example of fault information stored in the fault database. [Figure 4] Figure 2 shows an example of bug information stored in the bug database. [Figure 5] Figure 2 shows an example of test case information stored in the test case database. [Figure 6]It is a diagram showing an example of a result list generated by the output unit shown in FIG. 2. [Figure 7] It is an explanatory diagram showing the physical configuration of the calculation device. [Figure 8] It is a flowchart of the priority determination process by the calculation device.

Mode for Carrying Out the Invention

[0010] The calculation device, calculation method, and program according to the mode for carrying out the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals. Note that this embodiment is for the purpose of explanation and does not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each of these elements or all of these elements are replaced with equivalents thereof, and these embodiments are also included in the scope of the present invention.

[0011] (Overall Configuration) FIG. 1 is an explanatory diagram showing the cooperation between the calculation device 100 according to the embodiment of the present invention and other devices. As shown in the figure, the calculation device 100 is communicably connected to the terminal 200 via the communication network 300. In FIG. 1, one terminal 200 is shown, but the number of applicable terminals 200 is not limited to this, and a plurality of terminals 200 may be applied.

[0012] The calculation device 100 is composed of one or more server computers. The calculation device 100 is operated by, for example, an operator who develops software for various services provided through the Internet. Services provided through the Internet include, for example, online shopping malls, free market services, online supermarkets, reservation services such as accommodation facilities, Internet banks, electronic payment services, healthcare services, online learning, sports fan clubs, ticket sales, sales of voting tickets for public competitions, web searches, video distribution, music distribution, and the like. These services may be provided by server software of a website or by application software that operates on a terminal used by service users.

[0013] The calculation device 100 manages, for example, software tests in units of test targets. The test target unit may be a service provided through the Internet or a function realized by software in each service. Hereinafter, the management unit of the test is also referred to as a project. The calculation device 100 provides support to enable project participants to determine test cases that should be preferentially executed in a regression test by setting priorities for a plurality of test cases that can be executed in a project.

[0014] The calculation device 100 stores failure information about failures that occurred in the past, bug information indicating information about bugs discovered by executing test cases, test case information indicating information about test cases created for the purpose of verifying the operation and functions of software, verifying the modified parts after bug fixing, and the like.

[0015] The calculation device 100 estimates the probability of failure for each of the multiple test cases included in the regression tests planned to be performed in conjunction with the addition or modification of functions and the repair of defects, and sets the priority of each test case based on the estimated probability of failure. Specifically, the calculation device 100 estimates the probability of failure for each test case included in the target project based on the execution history of each test case and the execution history of test cases in other projects that are similar to the test case in question (also called similar test cases). For example, the calculation device 100 sets a higher priority for test cases that have a high estimated probability of failure.

[0016] Terminal 200 is an information terminal (so-called computer) such as a PC (Personal Computer), tablet, or smartphone, and is used by project participants in software development, such as QA engineers, software developers, and project managers. Terminal 200 is used, for example, to create specifications, design documents, programs, test cases, test plans, test results, test status, and register and view defect and bug information.

[0017] Furthermore, project participants operate terminal 200 to send a list of test cases that can be implemented in the target project to calculation device 100, and then view the resulting list generated by calculation device 100, which arranges the test cases for the target project in descending order of priority.

[0018] The communication network 300 may include various types of networks. For example, local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as public switched telephone networks (PSTNs), wireless networks, public switched networks, satellite networks, cellular networks, public land mobile communications networks (PLMNs), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic-based networks, etc., or any combination of these or other types of networks.

[0019] (Functional configuration of the calculation device) Figure 2 is an explanatory diagram showing the functional configuration of the calculation device 100. The calculation device 100 comprises a fault DB (Database) 110, a bug DB 120, a test case DB 130, a similarity calculation unit 140, an estimation unit 150, a priority determination unit 160, and an output unit 170.

[0020] The failure database 110 stores failure information related to failures that have occurred in the past. Figure 3 shows an example of failure information 111 stored in the failure database 110. As shown in the figure, failure information 111 includes information such as a "failure ID" to uniquely identify the failure information, a "service ID" to uniquely identify the service on which the failure occurred, a "project ID" to uniquely identify the project, a "failure details" indicating the nature of the failure, a "cause" indicating the cause of the failure, a "loss assessment" indicating the degree of loss due to the failure, a "correction method" indicating the correction made in response to the failure, and a "test case" indicating information that uniquely identifies the test case used when the failure occurred. Failure information 111 may also be generated, for example, based on the contents of a report after the failure occurred or a post-event report after the completion of a QA (Quality Assurance) task.

[0021] The "Fault Details" in Incident Information 111 may include information such as a detailed description of the fault, the circumstances and prerequisites under which the fault occurs, the procedure for reproducing the fault, the expected result showing the correct operation that should have been obtained, and the behavior when the fault occurred. In addition, the "Loss Assessment" may be an amount of damage indicating the economic loss or impact on business caused by the fault in monetary terms, or it may be information indicating the scope of impact such as the number of affected customers, the number of transactions, and the time, or it may be information ranking the degree of damage as S, A, B, C, etc.

[0022] The information included in the Failure DB110 is optional and may also include the failure title (which outlines the failure), the failure category, the date the failure was first reported, the status of the failure response, supplementary information such as log files, and feedback information.

[0023] Returning to Figure 2, the Bug DB 120 stores bug information about bugs discovered in the past. Bug information is created, for example, by the person who performed the test case when a bug is detected during the execution of the test case. Figure 4 shows an example of Bug Information 121. As illustrated, Bug Information 121 includes information such as a "Bug ID" to uniquely identify the bug information, a "Service ID" to uniquely identify the service in which the bug was discovered, a "Project ID" to uniquely identify the project, an "Event" indicating the content of the bug, an "Expected Result" indicating the correct operation that should be obtained, "Prerequisites" indicating the conditions under which the bug occurs, a "Procedure" indicating the steps to reproduce the bug, a "Target Function" indicating the function in which the bug occurs, a "Root Cause" indicating the cause of the bug, and a "Bug Type" indicating the category of the bug.

[0024] The information included in Bug DB120 is optional and may also include information such as the bug title which outlines the bug, the name of the person who discovered the bug and the date and time of discovery, test data which shows the data necessary to perform tests to verify the bug, the frequency of the bug's occurrence, and the environment in which the bug occurred, including the version information of the software in which the bug occurred.

[0025] Returning to Figure 2, the test case DB 130 stores test case information related to test cases. Figure 5 shows an example of test case information 131. As illustrated, test case information 131 includes information such as a "test case ID" to uniquely identify a test case, a "test case name" indicating the name of the test case, a "service ID" to uniquely identify the target service, a "project ID" to uniquely identify the target project, a "test procedure" indicating the specific steps for conducting the test, "test conditions" indicating the conditions that must be met before conducting the test, "expected results" indicating the expected behavior or results if the test is successful, "execution results" indicating the results when the test was actually conducted, "bug information" to identify bugs found in the test case, a "function name" indicating the function that the test case verifies, a "number of executions" indicating the number of times the test case has been conducted, a "number of failures" indicating the number of times the test case has failed, and "severity" indicating the degree of loss caused by the failure (including bugs and other defects) detected when the target test case fails. The "importance" field may store information ranked by the test case creator as S, A, B, C, etc., or it may store information ranked based on the impact of bugs discovered when the test case was previously executed, or the criticality of the problem.

[0026] Note that the information included in Test Case DB130 is optional and may also include information such as the test case title which outlines the test case, the name of the person who performed the test case and the date and time of performance, and the category of the test case.

[0027] Furthermore, if a test case has never been executed, information such as "bug information," "number of executions," "number of failures," and "severity" may be left blank and can be entered by the person in charge of the test case after it has been executed. Also, the "severity" value may differ between the time the test case is created and after it has been executed. For example, a test case may have its severity set to rank C when it was created, but after the test case is executed, a serious bug may be discovered, and the rank may be updated to rank S.

[0028] Returning to Figure 2, the similarity calculation unit 140 determines the similarity between each test case of the project being processed and test cases of other projects that have been implemented in the past. For example, the similarity calculation unit 140 calculates the similarity between each test case of the target project and test cases of other projects by methods such as numerically vectorizing the definition information of the test cases using any method such as word embeddings or BERT, and calculating the cosine similarity. The definition information includes text data such as the test case name, test procedure, and expected result for each test case. For each test case of the target project, the similarity calculation unit 140 extracts test cases of other projects whose calculated similarity is above a predetermined threshold as similar test cases. Details of the processing of the similarity calculation unit 140 will be described later.

[0029] The estimation unit 150 estimates the failure probability when each test case of the target project is executed. Specifically, the estimation unit 150 calculates the failure probability based on the execution history of the test case to be estimated and the execution history and similarity of similar test cases extracted by the similarity calculation unit 140 that are similar to the test case to be estimated. For example, the estimation unit 150 calculates the failure probability P of the test case to be estimated using the following formula 1. failure Calculate.

number

[0030] Note that in Equation 1, P failure,0P indicates the historical failure probability of the test case being estimated. failure,i w0 represents the historical failure probability of similar test case i. w0 also represents the weight of the test case being estimated, and w i This indicates the weight of the similar test case i. For example, if w0=1, then w i The value of may be set according to the similarity to the test case being estimated. Also, n represents the number of similar test cases.

[0031] The priority determination unit 160 determines the priority of each test case in the target project. For example, the priority determination unit 160 determines the priority based on predetermined determination rules, such as setting a higher priority according to the probability of failure of each test case estimated by the estimation unit 150.

[0032] Furthermore, the decision-making rules for determining priority are not limited to failure probability; for example, the frequency and timing of past test case executions may also be taken into consideration. For instance, if a test case is frequently executed or is scheduled to be executed soon, its priority may be increased.

[0033] The output unit 170 outputs a results list of test cases for the target project, sorted in descending order of priority. The output unit 170 includes, for example, a set maximum number of test cases in the results list. Figure 6 shows an example of a results list. As shown in the figure, the results list includes information such as the "priority" determined by the priority determination unit 160, the "test case name" indicating the name of the test case, and the "reason" indicating the reason why the test case was selected. Based on the priority determination process for each test case by the priority determination unit 160, the output unit 170 creates an explanatory text for the reason why the test case was selected, based on information such as the probability of failure, past execution frequency, and execution timing that serve as the basis for determining the priority, and inputs it into the "reason" field.

[0034] (Hardware configuration of the calculation device) Figure 7 is a block diagram showing the hardware configuration of the calculation device 100. The calculation device 100 includes a CPU 11 that executes processing according to a program, RAM 12 which is volatile memory, ROM 13 which is non-volatile memory, a storage unit 14 that stores data, an input unit 15 that accepts information input, a display unit 16 that visualizes and displays the information, and a communication unit 17 that sends and receives information, all of which are connected via an internal bus 99.

[0035] The CPU 11 controls the operation of the entire calculation device 100, is connected to each component, and exchanges control signals and data. The CPU 11 performs various processes by reading the program stored in the memory unit 14 into the RAM 12 and executing it. The CPU 11 performs the processes of the similarity calculation unit 140, estimation unit 150, priority determination unit 160, and output unit 170, which are the main functions provided by the program.

[0036] RAM12 is for temporarily storing data and programs, and holds programs and data read from memory unit 14, as well as other data necessary for communication. RAM12 is used as the work area of ​​CPU 11.

[0037] ROM13 stores control programs, BIOS (Basic Input Output System), etc., that the CPU11 executes for the basic operation of the calculation device 100.

[0038] The memory unit 14 includes a hard disk drive, flash memory, etc., and stores programs executed by the CPU 11, as well as various data used during program execution. The memory unit 14 functions as a fault database 110, a bug database 120, and a test case database 130.

[0039] The input unit 15 is a user interface equipped with a touch panel, keyboard, mouse, communication device, etc. The input unit 15 receives operation input from the user of the calculation device 100 and outputs a signal corresponding to the received operation input to the CPU 11.

[0040] The display unit 16 is a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display that visualizes and displays information.

[0041] The communication unit 17 is a network termination device or wireless communication device connected to a network, and a serial interface or LAN (Local Area Network) interface connected to them. The calculation device 100 communicates with the terminal 200 via the communication unit 17. The communication unit 17 functions as an output unit 170.

[0042] (Priority determination process) Next, the operation of the calculation device 100 will be explained with reference to Figure 8. This priority determination process starts, for example, when the calculation device 100 receives an instruction to start the priority determination process.

[0043] The similarity calculation unit 140 obtains a list of test cases that shows a list of multiple test cases for the project to be prioritized (step S101). Specifically, project participants operate the terminal 200 to specify the target project and create a list of test cases to be processed. For example, a project participant creates a list of test cases that displays the test case IDs of the target test cases in list format and sends it to the calculation device 100 along with the project ID. When the calculation device 100 receives the list of test cases, it proceeds to step S102.

[0044] In step S102, the similarity calculation unit 140 repeatedly executes the processes in steps S103 and S104 for all test cases in the list of test cases for the target project obtained in step S101.

[0045] In step S103, the similarity calculation unit 140 calculates the similarity between the test case to be processed and other test cases related to other projects (step S103). Specifically, the similarity calculation unit 140 calculates the similarity between the definition information of the test case to be processed and the definition information of other test cases related to other projects. The similarity calculation unit 140 refers to the test case DB 130 and uses definition information, including text data such as the test case name, test procedure, and expected result of each test case, to calculate the similarity using natural language processing techniques. For example, the similarity calculation unit 140 calculates the similarity between the test case of the target project and the test case of other projects by numerically vectorizing the definition information of the test case using any method such as word embeddings or BERT, and calculating the cosine similarity.

[0046] The definition information may include at least one of the test case name, test procedure, and expected result, or it may include other information stored in the test case information 131, or it may include bug information of the bug found in the test case (bug symptoms, expected result, prerequisites, procedure, root cause, bug type, affected function, etc.).

[0047] The similarity calculation unit 140 extracts test cases from other projects whose similarity is above a predetermined threshold as similar test cases and transmits them to the estimation unit 150. An upper limit may be set on the number of similar test cases extracted.

[0048] In step S104, the estimation unit 150 estimates the probability of failure if the test case to be processed is executed (step S104). Specifically, the estimation unit 150 calculates the probability of failure of the test case to be processed based on the execution history of the test case to be processed, the execution history of test cases from similar projects extracted in step S103, and the similarity calculated in step S103.

[0049] The estimation unit 150 refers to the test case information 131 illustrated in FIG. 5 stored in the test case DB 130, obtains the number of executions and the number of failures of the test case to be processed and similar test cases, and calculates the respective past failure probabilities P failure,0 and P failure,i Thereafter, the estimation unit 150 determines weights w0 of the test case to be processed and a weight w i corresponding to the similarity of the similar test case based on a preset rule. The estimation unit 150 substitutes the calculated P failure,0 , P failure,i , w0, and w i into Equation 1 to obtain the failure probability P failure of the test case to be processed.

[0050] Next, in step S105, the estimation unit 150 determines whether the process of loop 1 has been executed for all the test cases included in the test case list acquired in step S101. If the estimation unit 150 determines that there is an unprocessed test case, it executes the process of loop 1 for the unprocessed test case. On the other hand, if the estimation unit 150 determines in step S105 that the process of loop 1 has been executed for all the test cases included in the test case list, it proceeds to step S106.

[0051] Next, the priority determination unit 160 determines the priority of each test case of the target project based on a preset determination rule (step S106). For example, the priority determination unit 160 determines the priority based on a determination rule such as setting a high priority according to the height of the failure probability of the test case calculated by the estimation unit 150. Note that the determination rule may be set arbitrarily. For example, in addition to the failure probability, a rule that takes into account the execution frequency and execution timing of the test case may be used.

[0052] Next, the output unit 170 outputs a results list, as illustrated in Figure 6, in which the test cases for the target project are arranged in descending order of priority (step S107). For example, the output unit 170 extracts the set maximum number of test cases based on the priority set by the priority determination unit 160, and generates a results list by arranging them in descending order of priority. Based on information such as the failure probability, past execution frequency, and execution timing calculated in the priority determination process for each test case by the priority determination unit 160, the output unit 170 creates an explanatory text for the reason why the test case was extracted and inputs it into the "Reason" column. The output unit 170 may also create explanatory texts for test cases with higher priority and input them into the "Reason" column.

[0053] As described above, the calculation device 100 extracts similar test cases to each test case in the target project based on the definition information of each test case in the target project and other test cases related to other projects. The calculation device 100 calculates the failure probability of each test case in the target project based on the execution history of each test case in the target project and the execution history and similarity of similar test cases, and sets priorities according to the failure probability. Therefore, even for test cases with a small number of executions, the failure probability is estimated by taking into account the execution history of similar test cases, making it possible to set priorities more appropriately.

[0054] Furthermore, project participants can improve the efficiency of testing by referring to the priority of each test case set by the calculation device 100, thereby determining which test cases should be prioritized.

[0055] (modified version) In the above embodiment, the calculation device 100 was described as setting priorities based on the failure probability of each test case in the target project, but it is not limited to this. In addition to the failure probability, the calculation device 100 may also set priorities based on the degree of loss caused by the failure (including defects such as bugs) detected when a test case fails.

[0056] For example, in step S104, the estimation unit 150 may, in addition to calculating the failure probability, search the failure DB 110 to obtain the amount of damage stored in the failure information 111, which includes the test case ID of the test case to be estimated, as exemplified in Figure 3, and use the obtained amount of damage as the degree of loss for the test case to be estimated. Alternatively, the estimation unit 150 may refer to the test case information 131, which is exemplified in Figure 5, and calculate the degree of loss for the test case to be estimated based on the "importance" set for the target test case. If the "importance" is information ranked in predetermined stages such as S, A, B, C, etc., these may be converted to numerical values ​​to calculate the degree of loss. Note that the amount of damage and importance are examples of loss information.

[0057] Furthermore, if the estimation unit 150 is unable to calculate the extent of loss for reasons such as the absence of failure information 111 corresponding to the test case to be estimated, the failure information 111 not having a damage amount set, or the test case information 131 for the test case not having an importance level set, it may estimate the extent of loss for the test case to be estimated based on information associated with similar test cases. Specifically, the estimation unit 150 extracts similar test cases from the similarity calculation unit 140 that have a corresponding damage amount or importance level associated with them by referring to the corresponding failure information 111 or the test case information 131 for the similar test case. The estimation unit 150 identifies the damage amount or importance level of the extracted similar test cases. The estimation unit 150 may calculate the extent of loss for the test case to be estimated by a weighted average of the damage amounts or importance levels of the similar test cases based on the similarity level calculated in step S103.

[0058] In these cases, the priority determination unit 160 may determine the priority in step S106 based on the failure probability and the degree of loss calculated by the estimation unit 150, or it may calculate the expected loss by multiplying the failure probability and the degree of loss, and then determine the priority according to the calculated expected loss. This makes it possible to determine the priority by taking into account the degree of loss in addition to the failure probability of the test case, and to determine which test cases should be prioritized by more comprehensively evaluating the impact on the service to which the target project belongs.

[0059] Furthermore, if no similar test cases are extracted by the processing in step S103, and the number of tests performed on the test cases to be estimated is small (for example, the number of tests N=1 and the number of failures M=0), the estimation unit 150 may, when calculating the failure probability M / N in step S104, use an appropriate positive constant α (for example, α=0.5) to calculate the failure probability using the formula (M+α) / (N+α). This allows for a fairer comparison between test cases, even when there are differences in the number of tests performed on each test case, and prevents extreme probability values.

[0060] Furthermore, the output unit 170 may include information on bugs that may occur as a result of executing each test case in the results report illustrated in Figure 6. For example, the output unit 170 may refer to the "bug information" item in the test case information 131 of the target test case and include the bug IDs of bugs that have occurred in the past in the results report, or it may include the bug IDs of bugs that have occurred in the past in similar test cases whose similarity to the target test case is above a predetermined threshold in the results report.

[0061] Furthermore, the calculation device 100 according to the above embodiment can be implemented using a regular computer, not a dedicated device. For example, the calculation device 100 that performs the above processing may be configured by installing a program for performing any of the above-mentioned actions from a recording medium to a computer. Alternatively, the calculation device 100 may be configured by multiple computers working together.

[0062] Furthermore, if the above-mentioned functions are realized through a division of labor between the OS (Operating System) and the application, or through collaboration between the OS and the application, then only the parts other than the OS may be stored on the medium.

[0063] Furthermore, it is possible to superimpose a program onto a carrier wave and distribute it via a communication network. For example, the program could be distributed through an application store (App Store) or posted on a bulletin board system (BBS) on a communication network and distributed via the network. These programs can then be launched and executed under the control of the operating system, just like other application programs, to perform the aforementioned processing.

[0064] Furthermore, the information stored in the memory unit 14 is centrally managed by a cloud server located on the network, and the calculation device 100 may access the cloud server as needed to read and write information. In this case, the calculation device 100 does not need to have a fault DB 110, a bug DB 120, or a test case DB 130. Also, the candidate classification processing and priority determination processing by the calculation device 100 may be executed on the cloud using the information stored on the cloud server.

[0065] The various aspects of this disclosure are summarized below as an appendix.

[0066] (Note 1) A similarity calculation unit that determines the similarity between the definition information of each test case of a set of test cases related to a selected project and the definition information of other test cases related to other projects, An estimation unit estimates the probability that a new execution of each test case will fail, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the similarity calculated by the similarity calculation unit. A priority determination unit determines the priority of each of the test cases based on the probability estimated by the estimation unit, A calculation device equipped with [a specific feature].

[0067] (Note 2) The estimation unit estimates the probability by a weighted average based on the similarity between the past failure probabilities of each test case related to the selected project and the past failure probabilities of the other projects. The calculation device described in Appendix 1.

[0068] (Note 3) The estimation unit further estimates the degree of loss caused by the failure detected when each test case fails, based on the loss information included in the failure information associated with each test case related to the selected project, or the severity set for each test case. The priority determination unit determines the priority based on the probability and the degree of loss estimated by the estimation unit. The calculation device described in Appendix 1 or 2.

[0069] (Note 4) The estimation unit, if the loss information and importance level of the test case to be estimated are unavailable, extracts other test cases from among the other test cases to which the loss information or importance level is associated, and estimates the degree of loss of the test case to be estimated by a weighted average based on the similarity of the loss information or importance level of the extracted other test cases. The calculation device described in Appendix 3.

[0070] (Note 5) The aforementioned definition information includes at least one of the following: the name of the test case, the steps of the test case, and the expected result of the test case. The calculation device described in any one of the appendices 1 to 4.

[0071] (Note 6) The aforementioned similarity is the cosine similarity between a first vector, which is a vectorized representation of the definition information of the test cases related to the selected project, and a second vector, which is a vectorized representation of the definition information of other test cases related to the other projects. The calculation device described in any one of the appendices 1 to 5.

[0072] (Note 7) Computers For each of the multiple test cases related to the selected project, the step of determining the similarity between the definition information of each test case and the definition information of other test cases related to other projects, A step of estimating the probability that a new execution of each test case will fail, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the calculated similarity; The steps include determining the priority of each test case based on the estimated probability, A calculation method for performing this.

[0073] (Note 8) On the computer, For each of the multiple test cases related to the selected project, a process is performed to determine the similarity between the definition information of each test case and the definition information of other test cases related to other projects. A process for estimating the probability of a new execution of each test case failing, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the calculated similarity; A process to determine the priority of each test case based on the estimated probability, A program that executes the command.

[0074] This disclosure allows for various embodiments and modifications without departing from the broad spirit and scope of this disclosure. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of this disclosure. In other words, the scope of this disclosure is indicated by the claims, not by the embodiments. Various modifications made within the scope of the claims and the equivalent significance of the disclosure are considered to be within the scope of this disclosure. [Industrial applicability]

[0075] The present invention can be suitably employed to provide a calculation device, calculation method, and program capable of more appropriately determining the priority of test cases. [Explanation of symbols]

[0076] 100 Calculation device, 200 Terminal, 300 Communication network, 110 Fault DB, 120 Bug DB, 130 Test case DB, 140 Similarity calculation unit, 150 Estimation unit, 160 Priority determination unit, 111 Fault information, 121 Bug information, 131 Test case information, 11 CPU, 12 RAM, 13 ROM, 14 Storage unit, 15 Input unit, 16 Display unit, 17 Communication unit, 99 Internal bus

Claims

1. A similarity calculation unit that determines the similarity between the definition information of each test case of a set of test cases related to a selected project and the definition information of other test cases related to other projects, An estimation unit estimates the probability that a new execution of each test case will fail, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the similarity calculated by the similarity calculation unit. A priority determination unit determines the priority of each of the test cases based on the probability estimated by the estimation unit, A calculation device equipped with [a specific feature].

2. The estimation unit estimates the probability by a weighted average based on the similarity between the past failure probabilities of each test case related to the selected project and the past failure probabilities of the other projects. The calculation device according to claim 1.

3. The estimation unit further estimates the degree of loss caused by the failure detected when each test case fails, based on the loss information included in the failure information associated with each test case related to the selected project, or the severity set for each test case. The priority determination unit determines the priority based on the probability and the degree of loss estimated by the estimation unit. The calculation device according to claim 1 or 2.

4. The estimation unit, if the loss information and importance level of the test case to be estimated are unavailable, extracts other test cases from among the other test cases to which the loss information or importance level is associated, and estimates the degree of loss of the test case to be estimated by a weighted average based on the similarity of the loss information or importance level of the extracted other test cases. The calculation device according to claim 3.

5. The aforementioned definition information includes at least one of the following: the name of the test case, the steps of the test case, and the expected result of the test case. The calculation device according to claim 1 or 2.

6. The aforementioned similarity is the cosine similarity between a first vector, which is a vectorized representation of the definition information of the test cases related to the selected project, and a second vector, which is a vectorized representation of the definition information of other test cases related to the other projects. The calculation device according to claim 1 or 2.

7. Computers For each of the multiple test cases related to the selected project, the step of determining the similarity between the definition information of each test case and the definition information of other test cases related to other projects, A step of estimating the probability that a new execution of each test case will fail, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the calculated similarity; The steps include determining the priority of each test case based on the estimated probability, A calculation method for performing this.

8. On the computer, For each of the multiple test cases related to the selected project, a process is performed to determine the similarity between the definition information of each test case and the definition information of other test cases related to other projects. A process for estimating the probability of a new execution of each test case failing, based on the execution history of each test case related to the selected project, the execution history of the other test cases, and the calculated similarity; A process to determine the priority of each test case based on the estimated probability, A program that executes the command.

Images