Test case allocation method and computing device
By obtaining the historical execution time and execution status of test cases, it is divided into stable and non-stable classes, and grouped according to the estimated execution time, the problem of large differences in execution time between test environments is solved, and testing efficiency and stability are improved.
Patent Information
- Application Number
- CN202411096077.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-27
AI Technical Summary
During the existing software testing process, the allocation method of test cases leads to a large difference in execution time between test environments, resulting in inefficient testing.
By obtaining the historical execution time and execution status of the test case, determine its estimated execution time, and divide the test cases into stable and non-stable classes, grouping them according to the estimated execution time and categories, and allocating them to the test environment to control each test to control each test The difference between the total estimated execution time of the use case group and the expected execution time is less than or equal to the threshold.
It effectively improves the testing efficiency, makes the total estimated execution time of each test environment similar, reduces the instability of the execution time of the test case group, and avoids the problem of occupancy of the test environment for a long time.
Smart Images
Figure CN119201675A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of server technology, and in particular to a test case allocation method and computing device. Background Art
[0002] Software testing, as an important part of the software development process, is the process of discovering and correcting errors in the software. It is also an important means to ensure software quality, reliability, stability and user satisfaction.
[0003] In the current software testing process, test cases can be divided into levels, and each test case can be assigned to each test environment for testing according to the level of each test case. However, the level of test cases is usually divided based on the importance of the test case or the difficulty of execution. The execution time of test cases of different levels may be different, and the execution time between different test cases of the same level may also be different. Therefore, the method of assigning test environments to test cases according to their levels is prone to the situation where the total execution time in multiple test environments varies greatly. For example: when assigning test cases to test environments, multiple test cases with longer execution times may be assigned to test environment 1, and multiple test cases with shorter execution times may be assigned to test environment 2, so that the execution time of test environment 1 and test environment 2 differs greatly. The time consumption of the entire software testing process actually depends on the execution time corresponding to the test environment with the longest execution time. Therefore, the total execution time in multiple test environments differs greatly, which leads to low test efficiency. Summary of the invention
[0004] The embodiments of the present application provide a test case allocation method and a computing device, which can improve the execution efficiency of the test.
[0005] In the first aspect, an embodiment of the present application provides a test case allocation method, by obtaining a test task and a first expected execution time; the test task includes multiple test cases; obtaining the historical execution time and execution status of each test case in the multiple test cases; the execution status includes successful execution, failed execution and timed execution; determining the estimated execution time of each test case according to the historical execution time and execution status of each test case; determining multiple test case groups according to the estimated execution time; the difference between the sum of the estimated execution times of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to a first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group; and allocating the multiple test case groups to each of the test environments to execute the test task.
[0006] After the solution provided in the embodiment of the present application determines the estimated execution time of each test case, the difference between the sum of the estimated execution times of all test cases in each test case group and the second expected execution time corresponding to the test case group can be controlled to be less than or equal to a first threshold, so that the sums of the estimated execution times corresponding to each test case group are similar. In this case, each test case group is assigned to each test environment, and the total estimated execution time in each test environment is similar, which can effectively improve the overall testing efficiency.
[0007] In a possible implementation, the estimated execution time of each test case is determined based on the historical execution time and execution status of each test case. The method specifically includes: determining the category of the test case based on the difference between each historical execution time corresponding to each test case and the first expected execution time; the category of the test case includes: stable class or unstable class; according to the category and execution status of each test case, respectively determining the estimated execution time of the test case of the stable class and the test case of the unstable class.
[0008] In a possible implementation, the category of the test case is determined according to the difference between each historical execution time corresponding to each test case and the first expected execution time, including: when the number of differences between the historical execution time and the first expected execution time among the multiple historical execution times corresponding to each test case is less than or equal to the second threshold and greater than or equal to the third threshold, the category of the test case is determined to be a stable category, otherwise it is an unstable category. In the embodiment of the present application, the test cases are divided into stable and unstable categories, which is convenient for subsequent grouping and improving the overall execution efficiency.
[0009] In one possible implementation, the method for determining the estimated execution time of the test cases of the stable class includes: determining the proportion of successful executions in the execution status of the test cases of the stable class as the success rate of the test cases of the stable class; determining the test cases of the stable class whose success rate is greater than a fourth threshold as the successful stable class, otherwise determining them as the failed stable class; determining the average value of the historical execution time corresponding to the execution status of successful execution as the estimated execution time of the test cases of the successful stable class; determining the average value of the historical execution time corresponding to the execution status of failed execution as the estimated execution time of the test cases of the failed stable class. In the embodiment of the present application, the test cases of the stable class are divided into the test cases of the successful stable class and the test cases of the failed stable class, and their estimated execution times are determined respectively, which can improve the accuracy of the determined estimated execution time.
[0010] In one possible implementation, the method for determining the estimated execution time of the unstable test case includes: determining the first expected execution time as the estimated execution time of the unstable test case. In the embodiment of the present application, for the unstable test case, its execution time is unstable, and the first expected execution time is determined as the estimated execution time of the unstable test case. If the unstable test case is not completed within the estimated execution time during the actual execution, the execution of the test case can be stopped, and an error is reported or the test case is re-executed to avoid it occupying the test environment for a long time and reducing the overall test efficiency.
[0011] In one possible implementation, the determining of multiple test case groups based on the estimated execution time includes: when the category of the test case is a stable class, sorting the estimated execution time of the test cases of the stable class to obtain an estimated execution time sequence; using a first pointer and a second pointer to traverse the estimated execution time sequence from the head and the tail of the execution time sequence respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to a first threshold, determining that the test case corresponding to the first pointer and the test case corresponding to the second pointer constitute a test case group.
[0012] In one possible implementation, the method of determining multiple test case groups based on the estimated execution time includes: when the category of the test case is an unstable class, determining the number of test cases of the unstable class in each test case group based on the estimated execution time of the test cases of the unstable class and the second expected execution time; and grouping the test cases of the unstable class according to the number.
[0013] In one possible implementation, the allocating the multiple test case groups to each of the test environments to execute the test task includes: when the number of the test case groups is proportional to the number of the test environments, evenly allocating the multiple test case groups to each of the test environments to execute the test task; when the number of the test case groups is not proportional to the number of the test environments, evenly allocating a first number of the test case groups to each of the test environments, and allocating a second number of the test case groups to the test environment where the test group that completes the test first is located; wherein the sum of the first number of the test case groups and the second number of the test case groups is the number of the test case groups, thereby making the total estimated execution time in each test environment similar, which can effectively improve the overall test efficiency.
[0014] In a possible implementation, the multiple test case groups are allocated to each of the test environments to execute the test tasks, including: the unstable test case group and the stable test case group are allocated to different test environments to execute the test tasks. Since the unstable test case has a high probability of timeout execution and the execution time is unstable, in the embodiment of the present application, an exclusive execution method can be adopted, that is, for the unstable test case, a corresponding test environment is allocated to it, and the test environment can only be used to execute the unstable test case.
[0015] Second aspect: The embodiment of the present application provides a test case allocation device, which includes:
[0016] A first acquiring unit, a second acquiring unit, a first determining unit, a second determining unit, and an allocating unit;
[0017] The first acquisition unit is used to acquire a first expected execution time corresponding to a test task and a test case; the test task includes a plurality of test cases;
[0018] The second acquisition unit is used to acquire the historical execution time and execution status of each test case in the multiple test cases; the execution status includes successful execution, failed execution and timed execution;
[0019] The first determining unit is used to determine the estimated execution time of each test case according to the historical execution time and execution status of each test case;
[0020] The second determination unit is used to determine multiple test case groups according to the estimated execution time; the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group;
[0021] The allocating unit is used to allocate the multiple test case groups to each of the test environments to execute the test tasks.
[0022] Aspect 3: An embodiment of the present application provides a computing device, which includes a memory and a processor; the memory and the processor are coupled; the memory is used to store programs / instructions; the processor is used to call the programs / instructions for execution, so that the computing device implements a test case allocation method as described above.
[0023] Fourth aspect: An embodiment of the present application provides a program product. When the program product runs on a computer, the computer can execute a test case allocation method as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;
[0026] Figure 2 A flowchart of a test case allocation method provided in an embodiment of the present application;
[0027] Figure 3 A flow chart of a test case grouping method provided in an embodiment of the present application;
[0028] Figure 4 A flow chart of a method for grouping test cases of a stable class provided in an embodiment of the present application;
[0029] Figure 5 A schematic diagram of an allocation queue provided in an embodiment of the present application;
[0030] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] It should be noted that the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0032] In order to make the following embodiments clear, the technical terms involved in this application are first introduced.
[0033] A test case is a description of the testing task for a software product, reflecting the test plan, method, technology and strategy. Its content may include test objectives, test environment, input data, test steps, expected results, test scripts, etc. A test case can be considered as a set of test inputs, execution conditions and expected results compiled for a certain goal, used to verify whether a specific software requirement is met.
[0034] Testing Environment refers to the general term for the computer hardware, software, network equipment, and historical data required to complete software testing. It refers to the description of the software and hardware environment on which the test runs, as well as any other software that interacts with the software under test, including drivers and stubs.
[0035] The embodiment of the present application provides a test case allocation method, which obtains the historical execution time and execution status of each test case by statistically analyzing the historical test results of a large number of test cases. Obtain the first expected execution time corresponding to the test task and the test case; determine the estimated execution time of each test case based on the historical execution time and the corresponding execution status of each test case, and determine multiple test case groups based on the estimated execution time. Among them, the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group. On this basis, each test case group can be assigned to each test environment to execute the test task. After the estimated execution time of each test case is determined by the solution provided in the embodiment of the present application, the difference between the sum of the estimated execution times of all test cases in each test case group and the second expected execution time corresponding to the test case group can be controlled to be less than or equal to the first threshold, so that the sums of the estimated execution times corresponding to each test case group are similar. In this case, each test case group is assigned to each test environment, and the total estimated execution time in each test environment is similar, which can effectively improve the overall testing efficiency.
[0036] Combine the following Figure 1 This article introduces the application scenarios provided by this application.
[0037] like Figure 1 As shown in FIG. 1 , this figure is a schematic diagram of an application scenario provided by an embodiment of the present application. In the process of software development, when one or more of the functions, stability, and reliability of the developed software need to be tested, the software management device or management platform can be used to Figure 1 The computing device 1101 shown in the figure is connected to at least one server 1102 , wherein each server 1102 is equipped with a test environment, or a plurality of test environments are equipped on one server 1102 through virtual machines.
[0038] For example, when multiple test environments are mounted on a server 1102 through virtual machines, the test cases in each test environment mounted on the server 1102 can be executed independently, so that the test cases required for software testing can be executed in multiple test environments in parallel, thereby improving testing efficiency.
[0039] In a test task, the computing device 1101 usually needs to assign multiple test cases to the corresponding test environment for execution at least once. In order to improve the test efficiency, the computing device 1101 in the present application can obtain the historical test results of multiple test cases in the database, and determine the historical execution time and corresponding execution status of each test case in the aforementioned multiple test cases by statistically analyzing the historical test results of each test case in the multiple test cases, and determine the estimated execution time of the test case according to the historical execution time and the corresponding execution status of each test case. According to the estimated execution time, the aforementioned multiple test cases are grouped to determine multiple test case groups, so that the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold. The computing device 1101 assigns each test case group to the test environment on the server 1102 to perform the test task. Since the total estimated execution time in each test environment can be as similar as possible, the overall test efficiency is improved.
[0040] Based on the above description, a test case allocation method provided by an embodiment of the present application is introduced below. It can be understood that the method is proposed based on the above description, and part or all of the contents of the method can be referred to the above description. Figure 2 As shown, the figure is a flowchart of a test case allocation method provided in an embodiment of the present application, and the method can be executed by a computing device 1101.
[0041] S101, obtaining a first expected execution time corresponding to a test task and a test case.
[0042] The test task is a task initiated to execute a test case, and a test task contains multiple test cases. The first expected execution time is the expected execution time required to complete each test case, which can be set according to the requirements or can be the average of the historical execution time.
[0043] S102: Obtain the historical execution time and execution status of each test case in the multiple test cases.
[0044] When obtaining the test task and the first expected execution time, the computing device can determine multiple test cases that need to be executed in this test task from a large number of test cases stored in the database. For multiple test cases to be executed in this test task, the computing device can obtain the historical execution time and execution status of each test case from the database.
[0045] The historical execution time is the time taken to execute the test case once in the previously executed test tasks.
[0046] The execution status of a test case includes successful execution, failed execution, and timed execution. Successful execution means that the test case is executed within the expected time and the execution result is successful; failed execution means that the test case is executed within the expected time and the execution result is failed; timed execution means that the test case is not executed within the expected time.
[0047] For the test cases that are executed overtime, in order to avoid their execution time being too long, occupying the test environment for a long time, and affecting the overall test efficiency, if they are not completed within the maximum execution time, they will be stopped. It is understandable that the expected time and the maximum execution time allowed for the test case execution can be set according to the needs.
[0048] Exemplarily, test case A has been executed 4 times before this test process. The 4 historical execution times of test case A are t1, t2, t3, and t4. Among them, the execution status corresponding to t1 is successful execution, the execution status corresponding to t2 is successful execution, the execution status corresponding to t3 is failed execution, and the execution status corresponding to t3 is timed execution. In this case, the computing device can obtain the historical execution time and execution status of test case A in these 4 historical test tasks.
[0049] It should be noted that the execution status corresponding to the above-mentioned historical execution durations are only examples.
[0050] S103: Determine the estimated execution time of each test case according to the historical execution time and execution status of each test case.
[0051] In a possible implementation, since different test cases have different execution time stability, the computing device can determine the category of the test case according to the difference between each historical execution time corresponding to each test case and the first expected execution time. The categories of test cases include: stable category or unstable category. Compared with unstable category test cases, stable category test cases have higher actual execution time stability.
[0052] In one possible implementation, for each test case, when, among the multiple historical execution times corresponding to the test case, the difference between the historical execution time and the first expected execution time is less than or equal to the second threshold and the number is greater than or equal to the third threshold, the computing device determines that the category of the test case is a stable class; when the difference between the historical execution time and the first expected execution time is greater than the second threshold and the number is greater than or equal to the third threshold, the computing device determines that the category of the test case is an unstable class.
[0053] Exemplarily, the first expected execution time is 30, the second threshold is 5, and the third threshold is 8. The ten historical execution times corresponding to test case B are 29, 33, 35, 32, 28, 37, 25, 31, 34, and 30, respectively; the ten historical execution times corresponding to test case C are 29, 33, 37, 32, 28, 37, 25, 31, 24, and 30, respectively.
[0054] In this case, among the ten historical execution times of test case B, the historical execution times whose difference with the first expected execution time is less than or equal to the second threshold are 29, 33, 35, 32, 28, 25, 31, 34, and 30, respectively, for a total of 9. That is, the number of historical execution times whose difference with the first expected execution time is less than or equal to the second threshold is 9. Since 9 is greater than 8, the computing device determines that the category of test case B is a stable class.
[0055] Among the ten historical execution times of test case C, the historical execution times whose difference with the first expected execution time is less than or equal to the second threshold are 29, 33, 32, 28, 25, 31, and 30, respectively, for a total of 7. That is, the number of historical execution times whose difference with the first expected execution time is less than or equal to the second threshold is 7. Since 7 is less than 8, the computing device determines that the category of test case C is an unstable class.
[0056] It can be understood that in the embodiment of the present application, the values of the first expected execution time, the second threshold, and the third threshold are not specifically limited and can be set according to actual needs. The above are only examples.
[0057] Since the execution time stability of test cases of different categories is different, in order to improve the accuracy of the estimated execution time corresponding to each test case, after determining the category of each test case in the embodiment of the present application, the computing device calculates the estimated execution time of stable test cases and unstable test cases according to the category of each test case and the execution status corresponding to the historical execution time, and determines the estimated execution time corresponding to each test case.
[0058] In a possible implementation, for a test case of the stable class, the computing device determines the success rate of the test case according to the proportion of successful executions in the execution status of the test case of the stable class. Test cases with a success rate greater than a fourth threshold in the test case of the stable class are determined as successful stable classes, otherwise they are determined as failed stable classes.
[0059] For successful and stable test cases, the average value of the historical execution time corresponding to the successful execution status is used as the estimated execution time; for failed and stable test cases, the average value of the historical execution time corresponding to the failed execution status is used as the estimated execution time.
[0060] For example, when the fourth threshold is 70%, if the 10 historical execution times corresponding to test case D are T1, T2, ..., T10, and the execution statuses corresponding to T1 to T8 are all successful, then the proportion of successful executions is 80%, which is greater than 70%, and the computing device can determine test case D as a successful and stable test case; if the 10 historical execution times corresponding to test case E are T11, T12, ..., T20, and the execution statuses corresponding to T11 to T16 are all successful, and the execution statuses corresponding to T17 to T20 are all failed, then the proportion of successful executions is 60%, which is less than 70%, and the computing device can determine test case E as a failed and stable test case.
[0061] For the test case D of the successful stable class, its estimated execution time is T est =(T1+T2+……+T8) / 8; For the test case E of the failure stability class, its estimated execution time is T est =(T17+T18+…+T20) / 4.
[0062] For unstable test cases, the execution time is unstable, and the estimated execution time of the unstable test cases can be set according to actual needs. For example, the computing device can determine the first expected execution time as the estimated execution time of the unstable test cases.
[0063] If the non-stable test case is not completed within the estimated execution time when it is actually executed, the test case can be stopped and an error can be reported or the test case can be re-executed to avoid it occupying the test environment for a long time and reducing the overall test efficiency.
[0064] S104: Group multiple test cases according to the estimated execution time of each test case and the second expected execution time corresponding to the test case group, and determine multiple test case groups.
[0065] Among them, the estimated execution time corresponding to each test case group is the sum of the estimated execution times of the test cases included in each test case group. For example, if a test case group contains two test cases, then the estimated execution time corresponding to the test case group is the sum of the estimated execution times of the two test cases it contains. The second expected execution time is the time required to complete the execution of all test cases in a test case group, and its value is the product of the first expected execution time and the number of test cases in the test case group, wherein the first expected execution time and the number of test cases in the test case group can be set according to needs.
[0066] In the embodiment of the present application, among the multiple test case groups determined according to the estimated execution time of each test case, the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold value, thereby making the execution time required to complete all test cases in each test case group similar. Among them, the first threshold value can be set according to the needs, and the specific value of the first threshold value is not specifically limited in the embodiment of the present application.
[0067] In one possible implementation, compared with stable test cases, the actual execution time stability of unstable test cases is lower. In order to reduce the impact of unstable test cases on the execution efficiency of stable test cases, the stable test cases and unstable test cases can be grouped according to their estimated execution time to obtain multiple test case groups.
[0068] The following is an example of allocating two test cases in each test environment.
[0069] For unstable test cases, the computing device may determine the number of unstable test cases in each test case group according to the estimated execution time of the unstable test cases and the second expected execution time; and group the unstable test cases according to the number. The estimated execution time of the unstable test cases is the first expected execution time.
[0070] Exemplarily, if the second expected execution time is 60 and the first expected execution time is 30, then 2 of the unstable test cases in each test case group can be determined. In this case, the computing device can combine the unstable test cases in pairs to obtain multiple test case groups consisting of unstable test cases.
[0071] For stable test cases, in one embodiment, the computing device may sort the estimated execution times of the stable test cases to obtain an estimated execution time sequence. The estimated execution time sequence is traversed from the head and the tail of the execution time sequence using a first pointer and a second pointer respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to a first threshold, it is determined that the test cases corresponding to the first pointer and the test cases corresponding to the second pointer constitute a test case group.
[0072] In another embodiment, the stable test cases include successful stable test cases and failed stable test cases. When grouping the stable test cases, the computing device may group the successful stable test cases and failed stable test cases separately.
[0073] Exemplarily, the computing device sorts the estimated execution times of the test cases of the successful and stable class to obtain a sequence of estimated execution times corresponding to the test cases of the successful and stable class. The first pointer and the second pointer are used to traverse from the head and tail of the sequence of estimated execution times corresponding to the test cases of the successful and stable class, respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to the first threshold, it is determined that the test cases of the successful and stable class corresponding to the first pointer and the test cases of the successful and stable class corresponding to the second pointer constitute a test case group. Similarly, the computing device sorts the estimated execution times of the test cases of the failed and stable class to obtain a sequence of estimated execution times corresponding to the test cases of the failed and stable class. Use the first pointer and the second pointer to traverse from the head and tail of the estimated execution time sequence corresponding to the test cases of the failed stable class respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to the first threshold, determine that the test cases of the failed stable class corresponding to the first pointer and the test cases of the failed stable class corresponding to the second pointer constitute a test case group.
[0074] Thus, a test case group includes multiple successfully executed test cases, or includes multiple failed test cases.
[0075] Combine the following Figure 3 , and give an overall introduction to the above S103-S104. Figure 3 As shown, this figure is a flow chart of a test case grouping method provided in an embodiment of the present application. It should be noted that the embodiment of the present application does not specifically limit the number of test cases allocated to each test environment, and only two test cases are allocated to each test environment as an example for introduction.
[0076] S201, start calculating groups.
[0077] S202: Determine whether the category of the test case is a stable category.
[0078] The computing device determines the category of the test case according to the difference between each historical execution time corresponding to each test case and the first expected execution time.
[0079] For each test case, when, among the multiple historical execution times corresponding to the test case, the difference between the historical execution time and the first expected execution time is less than or equal to the second threshold and the number is greater than or equal to the third threshold, the computing device determines that the category of the test case is a stable class; when the difference between the historical execution time and the first expected execution time is greater than the second threshold and the number is greater than or equal to the third threshold, the computing device determines that the category of the test case is an unstable class.
[0080] If it is determined that the category of the test case is a stable category, S203 is executed to determine the estimated execution time of the test case of the stable category.
[0081] The computing device determines the success rate of the test case according to the proportion of successful execution in the execution status of the test case of the stable class. The test cases of the stable class whose success rate is greater than the fourth threshold are determined as the successful stable class, otherwise they are determined as the failed stable class.
[0082] For successful and stable test cases, the average value of the historical execution time corresponding to the successful execution status is used as the estimated execution time; for failed and stable test cases, the average value of the historical execution time corresponding to the failed execution status is used as the estimated execution time.
[0083] S204: Determine multiple test case groups according to the estimated execution time and the second expected execution time of the test cases of the stable class.
[0084] Combine the following Figure 4 A method for grouping test cases of a stable type provided in an embodiment of the present application is introduced.
[0085] S301. Sort the estimated execution times of stable test cases to obtain an estimated execution time sequence.
[0086] In one example, the computing device may sort the estimated execution times of all test cases that can be based on all stable classes to obtain an estimated execution time sequence.
[0087] Exemplarily, the estimated execution times of the 8 stable test cases are 30, 31, 33, 26, 30, 31, 31, and 27. The estimated execution times of the stable test cases are sorted to obtain an estimated execution time sequence of 26, 27, 30, 30, 31, 31, 31, and 33.
[0088] In another example, in the test cases of the stable class, the test cases of the stable class can be divided into the test cases of the successful stable class and the test cases of the failed stable class according to the success rate of the historical execution of the test cases. The computing device can sort the estimated execution time of the test cases of the successful stable class and the test cases of the failed stable class respectively, and obtain the estimated execution time sequence of the test cases of the successful stable class and the estimated execution time sequence of the test cases of the failed stable class, so that the test cases of the successful stable class and the test cases of the failed stable class can be grouped separately later.
[0089] For example, the estimated execution time of the test cases of the 8 stable classes are 30, 31, 33, 26, 30, 31, 31, and 27. Among them, 30, 31, 33, and 26 are the estimated execution time of the test cases of the successful stable class; 30, 31, 31, and 27 are the estimated execution time of the test cases of the failed stable class.
[0090] The estimated execution time of the test cases of the success and stability class are sorted, and the estimated execution time sequence of the test cases of the success and stability class is 26, 30, 31, 33; the estimated execution time of the test cases of the failure and stability class are sorted, and the estimated execution time sequence of the test cases of the failure and stability class is 27, 30, 31, 31.
[0091] It is understandable that in the embodiment of the present application, only the estimated execution time in the estimated execution time sequence is arranged in order from small to large as an example. Similarly, the estimated execution time in the estimated execution time sequence can also be arranged in order from large to small, which will not be repeated.
[0092] After obtaining the estimated execution time sequence, the following step S302 may be performed.
[0093] S302: Set two pointers, which may be respectively referred to as a first pointer and a second pointer.
[0094] The first pointer points to the head of the estimated execution time sequence, and the second pointer points to the tail of the estimated execution time sequence.
[0095] Take the case where the test cases of the successful and stable class and the test cases of the failed and stable class are grouped separately as an example. For the estimated execution time sequence of the test cases of the successful and stable class, the first pointer points to 26 and the second pointer points to 33; for the estimated execution time sequence of the test cases of the failed and stable class, the first pointer points to 27 and the second pointer points to 31.
[0096] It should be noted that when there are multiple minimum values, the order of the multiple minimum values can be arranged arbitrarily. After obtaining the estimated execution time sequence, the first pointer can first point to the minimum value ranked first; similarly, when there are multiple maximum values, the order of the multiple maximum values can be arranged arbitrarily. After obtaining the estimated execution time sequence, the second pointer can first point to the maximum value ranked last.
[0097] S303, calculating the sum of the estimated execution times pointed to by the first pointer and the second pointer, and determining the test case group according to the sum of the estimated execution times.
[0098] In the embodiment of the present application, after setting two pointers, the first pointer and the second pointer can be used to traverse the estimated execution time sequence from the head and the tail of the execution time sequence respectively, and calculate whether the sum of the estimated execution times pointed to by the two pointers is greater than the second expected execution time. The first pointer or the second pointer is moved according to whether the sum of the estimated execution times pointed to by the two pointers is greater than the second expected execution time, and after moving the pointers, the sum of the estimated execution times after the pointers are moved is calculated, that is, the sum of the estimated execution times pointed to by the current first pointer and the second pointer is calculated.
[0099] In one embodiment, when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to a first threshold, it is determined that the test case corresponding to the first pointer and the test case corresponding to the second pointer constitute a test case group.
[0100] In another embodiment, when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is minimal, it is determined that the test case corresponding to the first pointer and the test case corresponding to the second pointer constitute a test case group.
[0101] Exemplarily, whenever the sum of the estimated execution times is equal to the second expected execution time, the test case corresponding to the first pointer and the test case corresponding to the second pointer can be directly determined as a test case group.
[0102] Whenever the sum of the estimated execution times pointed to by the two current pointers is obtained, if the sum of the estimated execution times is equal to the second expected execution time, the test case corresponding to the first pointer and the test case corresponding to the second pointer can be directly determined as a test case group. If the sum of the estimated execution times is less than the sum of the second expected execution time and the first threshold, the first pointer is moved one position toward the tail, and the sum of the estimated execution times pointed to by the two current pointers is calculated again; if the sum of the estimated execution times is greater than the sum of the second expected execution time and the first threshold, the second pointer is moved one position toward the head, and the sum of the estimated execution times pointed to by the two current pointers is calculated again, until a test case group whose sum of the estimated execution times is closest to the second expected execution time is obtained, and the difference between the sum of the estimated execution times and the second expected execution time is less than or equal to the first threshold.
[0103] Take the estimated execution time sequence of the above-mentioned successful and stable test cases as an example. The estimated execution time sequence is: 26, 30, 31, 33. The first expected execution time is 30, the second expected execution time is 60, and the first threshold is 5. The first pointer points to 26, the second pointer points to 33, and the sum of the estimated execution times pointed to by the first and second pointers is 59. Since it is less than 60, the first pointer is moved one position toward the tail to point to 30. The sum of the estimated execution times pointed to by the current first and second pointers is 63. Since 59 is closer to 60 than 63, it is determined that the test cases corresponding to 26 and 33 form a test case group. By analogy, another test case group can be obtained, that is, the test cases corresponding to 30 and 31 form a test case group.
[0104] The first threshold represents the maximum difference between the sum of the estimated execution time of the acceptable test case group and the second expected execution time. In the process of determining the grouping, if the difference between the sum of the estimated execution time pointed to by the two pointers and the second expected execution time is greater than the first threshold, the possibility of the two test cases forming a test case group can be excluded.
[0105] By analogy, the test cases of the failed stable class can be grouped to obtain a test case group composed of the test cases of the failed stable class. Without distinguishing the test cases of the successful stable class and the failed stable class, all the test cases of the stable class can be grouped based on the above method to obtain multiple test case groups, which will not be repeated here.
[0106] Based on S202, when it is determined that the category of the test case is an unstable category, S205 can be used to determine the estimated execution time of the test case of the unstable category.
[0107] Therefore, the execution time of unstable test cases is unstable. In order to avoid the execution of unstable test cases affecting the overall test efficiency, in the embodiment of the present application, the first expected execution time is determined as the estimated execution time of the test case. If the test case is not executed within the first expected execution time, the execution of the test case can be directly terminated.
[0108] S206: Determine multiple test case groups according to the estimated execution time and the second expected execution time of the test cases of the unstable class.
[0109] In an embodiment of the present application, the computing device can determine the number of unstable test cases in each test case group based on the estimated execution time and the second expected execution time of the unstable test cases; and group the unstable test cases according to the number.
[0110] After determining the multiple test case groups, S105 may be performed to allocate the multiple test case groups to various test environments to execute the test tasks.
[0111] Since the estimated execution time corresponding to each test case group is close to the second expected execution time, that is, the estimated execution time corresponding to each test case group is approximate, the computing device can allocate multiple test case groups to each test environment according to the number of test environments and the number of test case groups, so that the total execution time in each test environment is as similar as possible, thereby improving the overall efficiency of the test.
[0112] In an embodiment of the present application, when the number of test case groups is proportional to the number of test environments, multiple test case groups can be evenly distributed to each test environment to execute the test task; when the number of test case groups is not proportional to the number of test environments, the first number of test case groups can be evenly distributed to each test environment, and the second number of test case groups can be distributed to the test environment where the test group that completes the test first is located.
[0113] The sum of the first number of test case groups and the second number of test case groups is the number of test case groups.
[0114] Exemplarily, if there are 3 test environments and 6 test case groups, at this time, the number of test case groups is proportional to the number of test environments, and the 6 test case groups can be evenly distributed to the 3 test environments, so that each test environment is assigned with two test case groups; if there are 3 test environments and 4 stable test case groups, at this time, the number of test case groups is not proportional to the number of test environments, and 3 test case groups can be evenly distributed to the 3 test environments, and the remaining test case group can be randomly assigned to any one of the 3 test environments, or the remaining test case group can be assigned to the test environment that completes execution first among the 3 test environments.
[0115] In a possible implementation, the unstable test case group and the stable test case group can be assigned to different test environments to execute the test tasks. Since the unstable test case has a high probability of timeout and the execution time is unstable, an exclusive execution method can be adopted, that is, for the unstable test case, a corresponding test environment is assigned to it, and the test environment can only be used to execute the unstable test case.
[0116] In one example, each test environment may correspond to a queue, and each queue is used to store a test case group assigned to the test environment. Figure 5 To understand, Figure 5 A schematic diagram of an allocation queue provided in an embodiment of the present application. A computing device may allocate a first queue, a second queue, and a third queue to a first test environment, a second test environment, and a third test environment, respectively, to perform test tasks, wherein the first queue, the second queue, and the third queue each contain at least one test case group.
[0117] In one example, the test case groups in each test environment can be executed in multiple threads to improve the test efficiency. Except for the different test cases to be executed in each test environment, other actions can be consistent. After the test cases in each test environment are executed, the test framework tool deployment, test case execution, test log collection, etc. can be performed.
[0118] In a possible implementation, after the test case is executed, the corresponding historical test results can be updated based on the test results of each test case, thereby ensuring the real-time nature of the historical test results. In this way, when a test task needs to be executed later, the computing device can accurately determine the estimated execution time of the test case based on the dynamically updated historical test results, so as to facilitate the subsequent grouping of the test cases, further ensuring the efficiency of the test.
[0119] In summary, in the embodiment of the present application, the test cases are grouped according to the estimated execution time determined by the historical execution time of each test case and the corresponding execution state, so that the difference between the estimated execution time corresponding to each test case group and the second expected execution time is less than or equal to the first threshold value. In this case, the estimated execution times between the test case groups are similar. Therefore, allocating multiple test case groups to each test environment to perform tests is equivalent to controlling the estimated execution time corresponding to each test environment, making the estimated execution time corresponding to each test environment similar, which can effectively improve the overall test efficiency.
[0120] like Figure 6 As shown, this figure is a structural schematic diagram of a computing device provided in an embodiment of the present application.
[0121] The computing device includes a memory 1200, a processor 1201 and a communication interface 1202; wherein the memory 1200 stores computer instructions, and the processor 1201 is used to execute the computer instructions, so that the computing device executes the test case allocation method shown above.
[0122] In some embodiments, the processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.
[0123] In some embodiments, the memory 1200 may be a volatile memory or a non-volatile memory, such as a register. Specifically, a volatile memory refers to a memory in which the data stored inside will be lost when the power supply is interrupted. Among them, the volatile memory is mainly a random access memory (RAM), including a static random access memory (SRAM) and a dynamic random access memory (DRAM). A non-volatile memory refers to a memory in which the data stored inside will not be lost even if the power supply is interrupted. Common non-volatile memories include read-only memory (ROM), optical disks, magnetic disks, solid-state hard disks, and various memory cards based on flash memory technology.
[0124] In some embodiments, the memory 1200 has an executable code, and the processor 1201 executes the code by obtaining a test task and a first expected execution time corresponding to a test case; the test task includes multiple test cases; the historical execution time and execution status of each test case in the multiple test cases are obtained; the execution status includes successful execution, failed execution and timed execution; according to the historical execution time and execution status of each test case, the estimated execution time of each test case is determined; according to the estimated execution time, multiple test case groups are determined; the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to a first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group; and the multiple test case groups are allocated to each test environment to execute the test task.
[0125] The communication interface 1202 may be used to implement communication between the computing device and the server. A test environment may be configured in the server to execute test cases.
[0126] The bus can be a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of understanding, Figure 6 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.
[0127] The embodiment of the present application provides a test case allocation device, and its specific implementation method is consistent with the implementation method and technical effects recorded in the embodiment of the above method, and some contents will not be repeated here.
[0128] The device comprises: a first acquisition unit, a second acquisition unit, a first determination unit, a second determination unit and an allocation unit;
[0129] The first acquisition unit is used to acquire a first expected execution time corresponding to a test task and a test case; the test task includes a plurality of test cases;
[0130] The second acquisition unit is used to acquire the historical execution time and execution status of each test case in the multiple test cases; the execution status includes successful execution, failed execution and timed execution;
[0131] The first determining unit is used to determine the estimated execution time of each test case according to the historical execution time and execution status of each test case;
[0132] The second determination unit is used to determine multiple test case groups according to the estimated execution time; the difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group;
[0133] The allocating unit is used to allocate the multiple test case groups to each of the test environments to execute the test tasks.
[0134] In a possible implementation, the first determination unit is specifically used to: determine the category of the test case based on the difference between each historical execution time corresponding to each test case and the first expected execution time; the category of the test case includes: stable class or unstable class; according to the category and execution status of each test case, determine the estimated execution time of the test case of the stable class and the test case of the unstable class respectively.
[0135] In a possible implementation, the first determination unit is specifically used to: when, among the multiple historical execution times corresponding to each test case, the difference between the historical execution time and the first expected execution time is less than or equal to the second threshold and the number is greater than or equal to the third threshold, determine that the category of the test case is a stable class; otherwise, it is an unstable class.
[0136] In a possible implementation, the first determination unit is specifically used to: determine the proportion of successful executions in the execution status of the test cases of the stable class as the success rate of the test cases of the stable class; determine the test cases of the stable class whose success rate is greater than a fourth threshold as the successful stable class, otherwise determine them as the failed stable class; determine the average value of the historical execution time corresponding to the execution status of successful execution as the estimated execution time of the test cases of the successful stable class; determine the average value of the historical execution time corresponding to the execution status of failed execution as the estimated execution time of the test cases of the failed stable class.
[0137] In a possible implementation, the first determination unit is specifically used to: determine the first expected execution time as the estimated execution time of the test case of the unstable class. In the embodiment of the present application, for the test case of the unstable class, its execution time is unstable, and the first expected execution time is determined as the estimated execution time of the test case of the unstable class. If the test case of the unstable class is not completed within the estimated execution time when it is actually executed, the execution of the test case can be stopped, and an error is reported or the test case is re-executed to avoid it occupying the test environment for a long time and reducing the overall test efficiency.
[0138] In a possible implementation, the second determination unit is specifically used to: when the category of the test case is a stable class, sort the estimated execution time of the test cases of the stable class to obtain an estimated execution time sequence; use a first pointer and a second pointer to traverse the estimated execution time sequence from the head and tail of the execution time sequence respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to a first threshold, determine that the test case corresponding to the first pointer and the test case corresponding to the second pointer constitute a test case group.
[0139] In a possible implementation, the second determination unit is specifically used to: when the category of the test case is an unstable class, determine the number of test cases of the unstable class in each test case group according to the estimated execution time of the test cases of the unstable class and the second expected execution time; and group the test cases of the unstable class according to the number.
[0140] In one possible implementation, the allocation unit is specifically used to: when the number of the test case groups is proportional to the number of the test environments, evenly allocate the multiple test case groups to each of the test environments to execute the test tasks; when the number of the test case groups is not proportional to the number of the test environments, evenly allocate a first number of the test case groups to each of the test environments, and allocate a second number of the test case groups to the test environment where the test group that completes the test first is located; wherein the sum of the first number of the test case groups and the second number of the test case groups is the number of the test case groups, thereby making the total estimated execution time in each test environment similar, which can effectively improve the overall testing efficiency.
[0141] In a possible implementation, the allocation unit is specifically used to allocate the unstable test case group and the stable test case group to different test environments to execute the test tasks. Since the unstable test case has a high probability of timeout execution and the execution time is unstable, in the embodiment of the present application, an exclusive execution method can be adopted, that is, for the unstable test case, a corresponding test environment is allocated to it, and the test environment can only be used to execute the unstable test case.
[0142] In summary, in the embodiment of the present application, the test cases are grouped according to the estimated execution time determined by the historical execution time of each test case and the corresponding execution state, so that the difference between the estimated execution time corresponding to each test case group and the second expected execution time is less than or equal to the first threshold value. In this case, the estimated execution times between the test case groups are similar. Therefore, allocating multiple test case groups to each test environment to perform tests is equivalent to controlling the estimated execution time corresponding to each test environment, making the estimated execution time corresponding to each test environment similar, which can effectively improve the overall test efficiency.
[0143] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a test case allocation method as described above are implemented.
[0144] An embodiment of the present application provides a program product. When the program product is run on a computer, the computer can execute the steps of the test case allocation method as described above.
[0145] The "first" and "second" in the names such as "first" and "second" mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.
[0146] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a disk, an optical disk, etc., including a number of instructions for a computing device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.
[0147] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0148] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. A test case allocation method, characterized in that: include: Obtaining a test task and a first expected execution time; the test task includes multiple test cases; Obtaining the historical execution duration and execution status of each test case in the multiple test cases; the execution status includes successful execution, failed execution, and timed execution; Determine the estimated execution time of each test case according to the historical execution time and execution status of each test case; Determining multiple test case groups according to the estimated execution time; The difference between the sum of the estimated execution time of all test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold, and the second expected execution time is the product of the first expected execution time and the number of test cases in the test case group; The multiple test case groups are allocated to each of the test environments to execute test tasks.
2. The method according to claim 1, characterized in that Determining the estimated execution time of each test case according to the historical execution time and execution status of each test case, the method specifically includes: Determine the category of the test case according to the difference between each historical execution time corresponding to each test case and the first expected execution time; the category of the test case includes: stable category or unstable category; According to the category and execution status of each test case, the estimated execution time of the stable test case and the unstable test case is determined respectively.
3. The method according to claim 2, characterized in that The step of determining the category of the test case according to the difference between each historical execution time corresponding to each test case and the first expected execution time includes: When, among the multiple historical execution times corresponding to each test case, the difference between the historical execution time and the first expected execution time is less than or equal to the second threshold and the number is greater than or equal to the third threshold, the category of the test case is determined to be a stable class, otherwise it is an unstable class.
4. The method according to claim 2, characterized in that: The method for determining the estimated execution time of the stable test case includes: The proportion of successful execution in the execution status of the test cases of the stable class is determined as the success rate of the test cases of the stable class; Determine the test cases of the stable class whose success rate is greater than a fourth threshold as a successful stable class, otherwise determine them as a failed stable class; The average value of the historical execution time corresponding to the execution status of successful execution is determined as the estimated execution time of the test cases of the successful and stable class; the average value of the historical execution time corresponding to the execution status of failed execution is determined as the estimated execution time of the test cases of the failed and stable class.
5. The method according to claim 2, characterized in that: The method for determining the estimated execution time of the non-stable test case includes: The first expected execution time is determined as the estimated execution time of the test case of the non-stable class.
6. The method according to claim 2, characterized in that The determining of multiple test case groups according to the estimated execution time includes: When the category of the test case is a stable class, the estimated execution time of the test cases of the stable class is sorted to obtain an estimated execution time sequence; The estimated execution time sequence is traversed from the head and the tail of the execution time sequence using a first pointer and a second pointer respectively, and when the difference between the sum of the estimated execution time corresponding to the first pointer and the estimated execution time corresponding to the second pointer and the second expected execution time is less than or equal to a first threshold, it is determined that the test case corresponding to the first pointer and the test case corresponding to the second pointer constitute a test case group.
7. The method according to claim 2, characterized in that: The determining of multiple test case groups according to the estimated execution time includes: When the category of the test case is an unstable class, determining the number of the test cases of the unstable class in each test case group according to the estimated execution time of the test cases of the unstable class and the second expected execution time; The test cases of the unstable class are grouped according to the number.
8. The method according to any one of claims 1 to 7, characterized in that: The allocating the plurality of test case groups to each of the test environments to execute the test tasks includes: In a case where the number of the test case groups is proportional to the number of the test environments, the multiple test case groups are evenly distributed to each of the test environments to execute the test tasks; When the number of the test case groups is not proportional to the number of the test environments, the first number of the test case groups are evenly distributed to each of the test environments, and the second number of the test case groups are distributed to the test environment where the test group that completes the test first is located; wherein the sum of the first number of the test case groups and the second number of the test case groups is the number of the test case groups.
9. The method according to any one of claims 1 to 8, characterized in that: The difference between the sum of the estimated execution times of all the test cases in each test case group and the second expected execution time corresponding to the test case group is less than or equal to the first threshold, including: the difference between the sum of the estimated execution times of all the test cases in each test case group and the second expected execution time corresponding to the test case group is the smallest.
10. A computing device, characterized in that The computing device includes a memory and a processor; The memory is coupled to the processor; The memory is used to store programs / instructions; The processor is used to call the program / instruction to execute, so that the computing device implements a test case allocation method as described in any one of claims 1-9.