A test case priority ordering method based on code changes and test history
By constructing a defect prediction model based on code changes and historical execution information, and combining it with inter-class relationship graphs and alternating sorting methods, the problem of ignoring code change information in existing technologies is solved, and more efficient test case priority sorting is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2023-08-29
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies ignore code change information in test case prioritization and require test cases to have sufficient historical execution length, failing to effectively utilize the differences between test cases.
A defect prediction model based on code changes and historical execution information is constructed using deep learning methods. Combined with inter-class relationship graphs and alternating sorting methods, different types of test cases are sorted, and multiple sorting strategies are integrated to improve the sorting effect.
It enables more accurate and efficient test case prioritization, reducing test time and resource waste.
Smart Images
Figure CN117130922B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of software testing in software engineering, and particularly relates to a method for prioritizing test cases based on code changes and test history. Background Technology
[0002] Test case prioritization refers to arranging test cases in software testing according to certain rules or standards, so that the testing team can execute tests more effectively with limited resources, reducing the waste of testing time and resources. Test case prioritization includes research based on coverage, change information, and deep learning methods.
[0003] Most existing research uses deep learning methods to build models that predict whether test cases can find defects. The most common strategy is to train defect prediction models based on historical execution information. However, existing solutions based on historical execution information use relatively simple information, ignore code change information, require test cases to have a sufficient historical execution length, and do not consider the differences between test cases. Summary of the Invention
[0004] This invention proposes a test case prioritization method based on code change and test history. First, a test case dataset based on code change information and historical execution information is collected. For each type of test case, a deep learning method is used to construct a defect prediction model. Then, based on the inter-class relationship graph, the source code affected by the code change is identified, and test cases for this source are selected. Test cases with recently discovered defects are categorized based on execution frequency and defect discovery status. Test cases with fewer historical executions and those with more executions and recently discovered defects are ranked according to the output of the corresponding prediction model and historical average execution time. Test cases with more executions and recently discovered defects are ranked according to their failure rate. Finally, the three sequences obtained from the classification and ranking are merged into a single test case sequence using an alternating ranking method. Experiments demonstrate that this method achieves better ranking results.
[0005] To achieve the above objectives, this invention employs a test case priority ranking method based on code changes and test history, comprising the following three main steps:
[0006] Step 1: Build defect prediction models for the two types of test cases respectively;
[0007] Step 1, which involves constructing defect prediction models for the two types of test cases, includes the following steps:
[0008] Step 1.1: For test cases that are executed infrequently, build a defect prediction model based on code change information. The model predicts whether the test cases can find defects based on the code change information.
[0009] For test cases that are executed a few times, because their historical execution count is low, it may not be possible to accurately predict whether defects can be found based solely on limited historical execution information. Therefore, a defect prediction model based on code change information can be trained based on the CodeBERT model. By learning the semantic relationship between test cases and code changes, it can be predicted whether test cases can find defects.
[0010] Step 1.2: For test cases that have been executed many times and have recently been found to have defects, build a defect prediction model based on historical execution information;
[0011] For this type of test case, information such as the historical execution results, average execution time, and number of methods contained in the test case can be obtained from the historical execution information of the continuous integration environment. These characteristics can be used to predict whether the test case can find defects.
[0012] Step 2: Select the source code affected by the code change based on the inter-class relationship diagram, select the test cases of this code and the test case set T of the recently discovered defects, sort the test cases in T to obtain three sequences S1, S2 and S3;
[0013] After a change is committed in a continuous integration environment, during testing, it is necessary to select the test cases affected by the change and determine the priority among the test cases. Therefore, by using static analysis and class relationship diagrams, all source code affected by the change is selected, and then test cases for these codes and test cases that have recently been found to have defects are selected. The test cases are then classified and sorted according to their characteristics.
[0014] Step 2 involves selecting source code affected by code changes based on the inter-class relationship diagram, selecting test cases for this code, and a set T of test cases with recently discovered defects. The test cases in T are then categorized and sorted to obtain three sequences S1, S2, and S3, including the following steps:
[0015] Step 2.1: Construct an inter-class relationship graph by statically analyzing the source code; select the source code affected by the code changes based on the inter-class relationship graph; select test cases for these codes; and select a set T of test cases for recently discovered defects.
[0016] Step 2.2: For test cases in T with fewer historical execution times, sort them in descending order using the output of the defect prediction model based on code change information as the first key and in ascending order using the historical average execution time as the second key to obtain sequence S1;
[0017] Step 2.3: For test cases in T that have been executed many times and have not been found to have defects in recent executions, a heuristic method is used to calculate their historical execution failure rate and sort them in descending order of failure rate to obtain sequence S2;
[0018] Step 2.4: For test cases in T that have been executed many times and have recently been found to have defects, sort them in descending order with the output of the defect prediction model based on historical execution information as the first key and sort them in ascending order with the historical average execution time as the second key to obtain sequence S3;
[0019] Step 3: Use the alternating sorting method to merge the three sequences S1, S2, and S3 obtained in Step 2 into a single sequence S;
[0020] Step 3, which uses an alternating sorting method to merge the three sequences S1, S2, and S3 obtained in step 2 into a single sequence S, includes the following steps:
[0021] Step 3.1: First, insert the test cases in sequence S3 whose predicted value is 1 into the beginning of sequence S.
[0022] In most cases, priority defect prediction models based on historical execution information are more accurate, so it is important to ensure that the test cases in these models have a higher priority than the test cases in the other two sequences.
[0023] Step 3.2: Insert the test cases in sequence S1 whose predicted value is 1 after the test cases added in step 3.1;
[0024] Step 3.3: Insert the remaining test cases in S1 and S3 after the test cases added in step 3.2.
[0025] Step 3.4: For sequence S2, insert it at the end of sequence S;
[0026] For test cases that have been executed many times and have not been found to have defects recently, the probability of them passing the most recent executions is high, and the probability of detecting defects is low. Therefore, they have the lowest priority in sequence S and need to be inserted at the end of sequence S.
[0027] The final priority sorted sequence S is obtained. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation
[0029] To more clearly illustrate the objectives and technical solutions of this invention, the specific embodiments of this invention will be described in more detail below with reference to the accompanying drawings.
[0030] To achieve the above objectives, this invention employs a test case priority ranking method based on code changes and test history, comprising the following three main steps:
[0031] Step 1: Build defect prediction models for the two types of test cases respectively;
[0032] This step uses deep learning methods to train defect prediction models for two types of test cases: one with fewer executions and the other with more executions and recently discovered defects. The model then predicts whether these two types of test cases can detect defects.
[0033] Step 1, which involves constructing defect prediction models for the two types of test cases, includes the following steps:
[0034] Step 1.1: For test cases that are executed infrequently, build a defect prediction model based on code change information. The model predicts whether the test cases can find defects based on the code change information.
[0035] The training defect prediction model refers to training a supervised learning model on the preprocessed RTT Torrent dataset by using code change information features to predict whether test cases can detect defects.
[0036] The code change information refers to the number of files modified in a single commit, the name of the most recently changed class, and the names of all changed files in a continuous integration environment.
[0037] Step 1.2: For test cases that have been executed many times and have recently been found to have defects, build a defect prediction model based on historical execution information;
[0038] The training defect prediction model refers to training a supervised learning model on the preprocessed RTT Torrent dataset based on historical execution information features to predict whether test cases can detect defects.
[0039] The historical execution information refers to the historical execution results of test cases in the regression testing method, the number of methods in the test cases, and the average execution time.
[0040] Step 2: Select the source code affected by the code change based on the inter-class relationship diagram, select the test cases of this code and the test case set T of the recently discovered defects, sort the test cases in T to obtain three sequences S1, S2 and S3;
[0041] This step constructs an inter-class relationship graph by statically analyzing the source code, then selects all source code affected by code changes based on the inter-class relationship graph, selects test cases for these codes, and divides these test cases into three categories according to the number of times the test cases are executed and the recent defect discovery situation, and designs a sorting method for each category.
[0042] Step 2 involves selecting source code affected by code changes based on the inter-class relationship diagram, selecting test cases for this code, and a set T of test cases with recently discovered defects. The test cases in T are then categorized and sorted to obtain three sequences S1, S2, and S3, which include the following steps:
[0043] Step 2.1: Construct an inter-class relationship graph by statically analyzing the source code; select all source code affected by code changes based on the inter-class relationship graph; select test cases for these codes and a set T of test cases for recently discovered defects;
[0044] Step 2.2: For test cases in T with fewer historical execution times, sort them in descending order using the output of the defect prediction model based on code change information as the first key and in ascending order using the historical average execution time as the second key to obtain sequence S1;
[0045] Step 2.3: For test cases in T that have been executed many times and have not been found to have defects in recent executions, a heuristic method is used to calculate their historical execution failure rate and sort them in descending order of failure rate to obtain sequence S2;
[0046] The historical execution failure rate refers to the HFR (Formula 1) of the test cases, where and |TH i (t)| represents the number of times test case t detected defects and the total number of historical executions in the i-th integration cycle, respectively.
[0047]
[0048] Step 2.4: For test cases in T that have been executed many times and have recently been found to have defects, sort them in descending order with the output of the defect prediction model based on historical execution information as the first key and sort them in ascending order with the historical average execution time as the second key to obtain sequence S3;
[0049] Step 3: Use the alternating sorting method to merge the three sequences S1, S2, and S3 obtained in Step 2 into a single sequence S;
[0050] This step merges the three sequences from step 2 into a single sequence using an alternating sorting method.
[0051] Step 3, which uses an alternating sorting method to merge the three sequences S1, S2, and S3 obtained in step 2 into a single sequence S, includes the following steps:
[0052] Step 3.1: Insert the test cases in sequence S3 whose predicted value is 1 into the beginning of sequence S;
[0053] For example, the S3 sequence is in The test case has a predicted value of 1. If the test case has a predicted value of 0, then the sequence S is:
[0054] Step 3.2: Insert the test cases with a predicted value of 1 in sequence S1 after the test cases added in step 3.1;
[0055] For example, the S1 sequence is in The test case has a predicted value of 1. If the test case has a predicted probability of 0, then the sequence S is...
[0056] Step 3.3: Insert the remaining test cases in S1 and S3 after the test cases added in step 3.2.
[0057] After adding the remaining test cases from sequences S1 and S3 to sequence S, sequence S becomes...
[0058] Step 3.4: For sequence S2, insert it at the end of sequence S;
[0059] If the sequence S2 is {t2}, then the sequence S is
[0060] The final priority sorted sequence S is obtained.
Claims
1. A test case priority ranking method based on code changes and test history, characterized in that, Includes the following steps: Step 1: Construct defect prediction models for the two types of test cases, including the following steps: Step 1.1: For test cases that are executed infrequently, construct a defect prediction model based on code change information; Step 1.2: For test cases that have been executed many times and have recently been found to have defects, build a defect prediction model based on historical execution information; Step 2: Based on the inter-class relationship diagram, select the source code affected by the code changes, and select test cases for this code, as well as a set of test cases for recently discovered defects. ,right The test cases were sorted by category to obtain three sequences. , , This includes the following steps: Step 2.1: Construct an inter-class relationship graph through static analysis of the source code. Based on the inter-class relationship graph, select the source code affected by code changes, and select test cases for this code, as well as a set of test cases for recently discovered defects. ; Step 2.2, for Test cases with fewer historical execution counts are sorted in descending order using the output of a defect prediction model based on code change information as the first key, and in ascending order using the historical average execution time as the second key to obtain the sequence. ; Step 2.3, for For test cases that have been executed frequently and for which no defects have been found in recent executions, a heuristic method is used to calculate their historical failure rate, and then the test cases are sorted in descending order of failure rate to obtain a sequence. ; Step 2.4, for Test cases that have been executed frequently and whose defects were recently discovered are sorted in descending order using the output of a defect prediction model based on historical execution information as the first key, and in ascending order using the historical average execution time as the second key to obtain the sequence. ; Step 3: Use the alternating sorting method to sort the three sequences obtained in Step 2. , , Merge into a sequence This includes the following steps: Step 3.1, first, the sequence Test cases whose predicted value is 1 are inserted at the beginning of sequence S; Step 3.2, sequence Test cases with a predicted value of 1 from the model are inserted after the test cases added in step 3.1; Step 3.3, sequentially... and The remaining test cases are inserted after the test cases added in step 3.2; Step 3.4, targeting Sequence, insert it into sequence The tail section; The final priority sorted sequence is obtained. .