Automatic patch filtering method and system based on historical version test oracle
By inserting the test framework into the defect history version to obtain the test prediction, the problem of difficult to obtain and generate too much input space in the existing technology is solved, and efficient patch filtering and test generation are achieved.
Patent Information
- Application Number
- CN202510463142.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-07-29
AI Technical Summary
The existing patch filtering technology is difficult to obtain test predictions in dynamic methods, and the static method is inefficient, the defective data set lacks historical information, and the test generation input space is too large, making it difficult to generate effective unanswered tests.
By inserting the test framework into the historical version of the defect, obtain the test predictions, use the binary search technology of the Defects4j framework and Git repository to collect the historical version, combine the data extraction and path constraint extraction of the unblocked test case, generate new test cases, and execute them in the historical version and the current version to obtain the test predictions and results, and compare and filter and verify.
Improve the reliability and effectiveness of patch filtering technology, and the test prediction verification patches provided by historical versions enhance the accuracy and efficiency of test generation.
Smart Images
Figure CN120386730A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of software engineering and software testing, and in particular to a method for automatically filtering program defects patches. Background Art
[0002] 1. Patch filtering technology
[0003] For existing patch filtering technologies, dynamic methods have higher accuracy, but they often face the problem of difficulty in obtaining test predictions when generating new tests, while static methods have high efficiency, but they have to face the problem of poor versatility. Existing patch filtering methods often only focus on the differences in various program states between defective versions and patched versions, while the actual development process is an iterative process, and each code submission may introduce new defects. This means that the code changes in a historical version where the defect exists can help us understand the changes in the program before and after the defect is introduced. Before the current defect is introduced, there may be a correct version that retains the complete functionality of the function where the defect is located. Therefore, how to implement a patch filtering technology that retains both the accuracy of the dynamic method and some of the efficiency of the static method is a technical problem that needs to be solved urgently in the present invention.
[0004] 2. Test enhancement technology
[0005] In software engineering, test enhancement techniques aim to improve software reliability by generating test cases with wider coverage and more diverse scenarios. The test prediction problem involves determining the consistency between the actual and expected outputs of a test case to verify the correct behavior of the software under test. This problem not only determines the credibility of test results but also directly impacts the degree of test generation automation. In practical applications, the test prediction problem faces numerous challenges, such as the high cost of manually defining expected outputs and the inaccuracy of output definitions for complex systems.
[0006] However, several important challenges remain in achieving test enhancement.
[0007] (1) The existing defect dataset does not contain historical information. A correct version of the code without the defective functionality is required, but the large number of historical versions of the defective code and the refactoring issues that occur during the evolution process make it difficult to determine the code version that introduced the defect. To this end, the Defects4j framework is used to integrate code history information to complete the automatic build and execution process, and manual inspection is performed for defects that are difficult to determine.
[0008] (2) The generated input space for testing is too large. Due to the lack of strong pertinence of existing test generation tools to specific defects and the long time-consuming generation, it is difficult to generate error-revealing tests. Using the failed tests that trigger defects first restricts the execution path of the generated tests, reduces the generation space of test case inputs, and thus enhances the error-revealing ability of the generated tests. Summary of the Invention
[0009] The present invention aims to propose an automatic patch filtering method and system based on mining test oracles from historical versions, and realizes an automatic patch filtering technology for obtaining test oracles by inserting an enhanced test framework after the historical versions of defects.
[0010] The present invention is realized by the following technical solutions:
[0011] In a first aspect, an automatic patch filtering method based on test oracles of historical versions proposed by the present invention includes:
[0012] Collect a dataset of code defects and version builds;
[0013] Extract error-revealing test case data according to the dataset to obtain a test framework for error-revealing test results;
[0014] Based on the test framework, implement: obtain newly generated test cases, execute the newly generated test cases on the historical version and the current version respectively, obtain test oracles and test results, compare the test oracles and the test results, and obtain patch filtering verification.
[0015] In some embodiments, the collection of code defects and their historical versions further includes:
[0016] The collection of code defects and their historical versions further includes automatically finding the historical versions of each defect: extracting the running attributes of the defect code in the current version, extracting the running attributes of the defect code in the historical version, executing a script file, and searching forward for the corresponding historical version.
[0017] In some embodiments, extracting bug-finding test case data according to the dataset further includes purifying, splitting, and path constraint extraction of bug-finding test cases; wherein, the purification includes deleting all statements after the statement that triggers failure in the bug-finding test case and redundant test assertions, and each bug-finding test function has exactly one test assertion; the splitting includes splitting the statements of the purified bug-finding test case into three parts: literal variable definition, test assertion, and other statements, obtaining new test inputs by modifying the content of the defined variables, and obtaining the test framework for bug-finding tests; the path constraint extraction includes extracting the functions covered in the execution path of the bug-finding test case from the bug-finding test case, analyzing the specifically covered statements through the function call relationship and extracting the constraint conditions in the statements, and constructing the execution path of the bug-finding test case according to the constraint conditions.
[0018] In some embodiments, modify the test assertion part in the newly generated test case to an executable result, output it to the output statement saved by the specified file, and then place the newly generated test case into the corresponding historical version for execution to obtain the test oracle.
[0019] In some embodiments, further executing the test framework includes randomly generating test cases as test inputs, performing random mutation according to the variable types obtained in the data extraction and the constraint conditions in the execution path, using the mutated test inputs as the mutated test inputs, and putting the mutated test inputs back into the split bug-finding test case to obtain the newly generated test case.
[0020] In some embodiments, after obtaining the test oracle, place the newly generated test case into the patch version for execution to obtain the test results in the patch version.
[0021] In a second aspect, an automatic patch filtering system based on a historical version test oracle proposed by the present invention includes a construction module, an extraction module, and a test framework module; the construction module is used to construct a dataset of code defects and versions collected; the extraction module is used to extract bug-finding test case data according to the dataset to obtain the test framework for the bug-finding test results; the test framework module is used to implement based on the test framework: obtaining newly generated test cases, executing the newly generated test cases in the historical version and the current version respectively through the newly generated test cases, obtaining the test oracle and the test results, comparing the test oracle and the test results, and performing patch filtering verification.
[0022] Generating a test framework for bug-finding tests for the patch program through test enhancement technology improves the reliability;
[0023] Using the historical version to provide the test oracle and applying it to the patch filtering task verification improves the effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flowchart of an automatic patch filtering method based on historical version test predictions of the present invention;
[0025] Figure 2 is a module diagram of an automatic patch filtering system based on historical version test predictions of the present invention;
[0026] Figure 3 is a technical roadmap. DETAILED DESCRIPTION OF THE INVENTION
[0027] The following will further describe in detail the specific embodiments of the present invention with reference to the accompanying drawings.
[0028] As Figure 1 shown, an automatic patch filtering method based on historical version test predictions of the present invention
[0029] Step 1, collect code defects and their historical versions to construct a data set; wherein the data set includes defects and historical versions of the defects;
[0030] 1.1, automatically find the historical versions of each defect; the specific formal definition is as follows:
[0031] Define the defect as b i , the total number of defects is m, and the code version library corresponding to this defect b i is V i ={v0, v1,... v k , v k+1 ,... v n , v n+1}, where n + 2 is the total number of versions, v n is the defect version, and v n+1 is the fixed version. Assume that the set of error-revealing test cases T i triggering this defect b i executes successfully on the previous version v k and fails on the next version v k+1 . In this case, the previous version v k is used as the historical version V i of the defect b historyi . Iteratively test each code version of the defect b i in this way, and the first v k obtained is used as the historical version of the defect b i . Perform the above operations on m defects to construct the historical version library V history ={V history1 , V history2 ,... V historyi …, Vhistorym , where m is the total number of historical versions;
[0032] Automatically find the historical version V corresponding to each defect historyi The specific implementation process includes:
[0033] 1.1.1. Use the Defects4j framework to execute relevant properties through the Defects4j command, including but not limited to trigger.test, etc., and write a script file write_profile.pl so that the Defects4j command can be executed in different versions of the defect; among them, the script write_profile.pl mainly includes three parts: 1) Extract the defect-related properties of the existing defect version code (including the class names involved and the name of the error-revealing test function); 2) Extract the running properties of the historical version code, and the running properties are mainly extracted from the build file of the code itself (including build.xml or pom.xml file); 3) Generate a new profile file, including the running properties required for the Defects4j framework to execute compilation and test tasks on the code of this version; execute the script file to extract the running properties of each defect;
[0034] 1.1.2. Combine the binary search command (git bisect) of the Git repository to search forward for the corresponding historical version, and collect the defects that can find the historical version into the dataset; the process of searching forward for the corresponding historical version is specifically described as follows:
[0035] i) First, start binary testing on the fixed version: Use the method of git log --pretty=format:%H > commits to store the code version numbers sha of all versions before the current version in the commit information (commits) file, where the first line content is the sha of the current version, and the last line is the first commit of this repository (i.e., initial_commit), and then use the git bisect start HEAD $initial_commit command to limit the binary test range to the code version from the initial commit to the fixed version, so as to perform binary testing;
[0036] ii) Execute `git bisect start` to begin the binary search test. After starting, it will randomly jump to the code in a version within a certain range and start checking. Copy the fault - revealing test cases and execute the fault - revealing test cases using the test command `defects4j test` of the Defects4j framework. If the code build of this version fails, the current binary search test will stop automatically. Otherwise, after verifying that the fault - revealing test cases are executed successfully, execute the `git bisect good` command to mark the current version as the code version before the defect was introduced. If the verification test result is a failure, execute the `git bisect bad` command to mark the current version as the code version after the defect was introduced.
[0037] iii) If the current binary search test has not stopped, the above - mentioned binary search test process in ii) will be continuously repeated until the result is output after the binary search test is completed. If the result is finding the first code version where the fault - revealing test fails ($sha is the first bad commit), then the code version represented by this $sha is the version v where the defect was introduced. k+1 , and its previous version is the required historical version v k ; Otherwise, mark the current defect as not finding the historical version and do not add it to the dataset.
[0038] 1.2, Generate test oracles for each defect based on the historical version. Assume that the fault - revealing test cases behave the same on two versions v k , v k+1 . This indicates that the function of the defect in the corresponding version can work properly in the historical version, and generate the test results of the fault - revealing test cases as test oracles for the defect.
[0039] Step 2, Extract the fault - revealing test case data based on the dataset in Step 1 to obtain the test framework for the fault - revealing test results. This step further includes purifying, splitting, and extracting path constraints for the fault - revealing test cases:
[0040] 2.1, Fault - revealing test purification is to purify the fault - revealing test cases, that is, the test cases that trigger failures (trigger.test), including deleting all statements after the statement that triggers the failure and redundant assertion statements in the fault - revealing test cases. The purification of the fault - revealing test cases ensures that each fault - revealing test case function has exactly one test assertion statement. Through the purification operation, a set of refined unit fault - revealing tests are generated from each original fault - revealing test case.
[0041] 2.2. Split the purified fault - revealing test cases, that is, split the statements of the purified fault - revealing test cases into three parts: literal variable definitions, test assertions, and other statements. Among them, the literal variable definitions are separate variable definitions extracted according to the types of literal parts on which the assertion statements depend (including byte, short, int, long, char, double, float, their corresponding wrapper types, and the String type). This variable will be used as the input for the newly generated test cases later. Obtain a new test input by modifying the content of the above - defined variable, thereby obtaining the test framework for the fault - revealing test. Specifically, if there is a literal in the execution part of the test assertion, then this literal is the required variable definition; if the execution part only contains non - literal variables, perform dependency analysis, obtain all the literal contents on which the variable depends and add them to the list of required variable definitions, and keep the rest unchanged; generating the input of the fault - revealing test cases more conveniently through the splitting operation;
[0042] 2.2. Extract path constraints. Extract the function information covered in the execution path of the fault - revealing test cases from the fault - revealing test cases, that is, the test cases that trigger defects, including the function call chain obtained from the assertion statement of the fault - revealing test case to the statement position where the defect is triggered. Analyze the specific covered statements through this function call relationship and extract the constraint conditions in the statements, and construct the execution path of the fault - revealing test cases with the analyzed constraint conditions. Conduct a specific analysis of each function in the call chain, analyze the list of variables on which the function depends according to the constant list obtained after splitting, and extract the involved path constraint conditions (such as the conditional expressions of if statements, while / for statements, etc.), thereby constructing the constraint relationship of each function. Further narrow the generation space of test inputs through path constraint extraction and increase the possibility of triggering defects;
[0043] Step 3. Execute the test framework to obtain different newly generated test cases. That is, according to the list of variable definitions and path constraint conditions obtained in Step 2, perform random mutation. Specifically, for each variable in the list of variable definitions, randomly replace it with other values of the same type according to its variable type, and restrict it to satisfy the path constraint conditions, and use it as the test input of the mutated test case. Then put the mutated variable definitions back into the split fault - revealing test cases to obtain different newly generated test cases. This step expands the test cases applicable to different defects through test generation;
[0044] Step 4: Obtain the test oracle by generating new fault - revealing test cases, that is, modify the test assertion part in the newly generated fault - revealing test cases obtained in Step 3 to the executable result, output it to the output statement saved in the specified file, and then place the newly generated fault - revealing test cases into the corresponding historical version for execution, so as to obtain the test oracle; through this step, it is more convenient to obtain the test oracle.
[0045] Step 5: Perform patch filtering verification by obtaining the test oracle, that is, after obtaining the test oracle, put the newly generated test cases into the patch version for execution, use the execution results of different test cases in the patch version as the test results, and use the test oracles obtained from the historical versions of each defect one - to - one as the conditions for verifying the patch; since it is considered that the test oracles in the historical versions are correct, the filtering result is that the test cases that fail to execute in the patch version are fault - revealing tests, and the patches with fault - revealing tests are incorrect patches.
[0046] As Figure 2 As shown, the automatic patch filtering system based on the historical version test oracle of the automatic patch filtering method based on the historical version test oracle of the present invention further includes a construction module 100, an extraction module 200, and a test framework module 300. Among them: the construction module 100 is used to collect code defects and their historical versions and construct a data set. The extraction module 200 is used to extract fault - revealing test case data according to the data set to obtain a test framework for the fault - revealing test results. The test framework module 300 is used to execute the test framework, obtain newly generated test cases, obtain the test oracle through the newly generated fault - revealing test cases, and perform patch filtering verification by obtaining the test oracle.
[0047] In summary, the present invention realizes a test generation framework for mining test oracles based on historical versions, and uses this test generation framework to realize an automated method for finding historical versions. First, complete the data set construction part. After collecting the historical versions of defective codes, successively perform four parts: data extraction, test generation, test oracle acquisition, and patch verification.
[0048] It should be noted that although the present invention has been shown and described with reference to specific exemplary embodiments of the present invention, those skilled in the art should understand that the present invention is not limited to the above - mentioned embodiments. Any changes to the present invention fall within the protection scope of the present invention application.
Claims
1. An automatic patch filtering method based on historical version test oracles, characterized in that The method includes: collecting the collected code defects and version build datasets; extracting bug-finding test case data according to the dataset to obtain a test framework for bug-finding test results; implementing based on the test framework: obtaining newly generated test cases, executing the newly generated test cases in the historical version and the current version respectively, obtaining test oracles and test results, comparing the test oracles and the test results, and obtaining patch filtering verification.
2. The automatic patch filtering method based on historical version test oracles according to claim 1, wherein The further collection of code defects and their historical versions includes automatically finding the historical versions of each defect: extracting the running attributes of the defect code in the current version, extracting the running attributes of the defect code in the historical version, executing the script file, and searching forward for the corresponding historical version.
3. An automatic patch filtering method based on a historical version test prediction according to claim 1, characterized in that, The extraction of bug-finding test case data according to the dataset further includes bug-finding test case purification, splitting, and path constraint extraction; wherein, the purification includes deleting all statements after the statement that triggers failure in the bug-finding test case and redundant test assertions, and each bug-finding test function has and only has one test assertion; the splitting includes splitting the statements of the purified bug-finding test case into three parts: literal variable definition, test assertion, and other statements, obtaining new test inputs by modifying the content of the variables defined above, and obtaining a test framework for bug-finding tests; the path constraint extraction includes extracting the functions covered in the execution path of the bug-finding test case from the bug-finding test case, analyzing the specifically covered statements through the function call relationship and extracting the constraint conditions in the statements, and constructing the execution path of the bug-finding test case according to the constraint conditions.
4. An automatic patch filtering method based on a historical version test prediction according to claim 3, characterized in that, Modify the test assertion part in the newly generated test case to an executable result, output it to the output statement saved by the specified file, and then place the newly generated test case in the corresponding historical version for execution to obtain the test oracle.
5. An automatic patch filtering method based on historical version test oracles according to claim 1, characterized in that The execution of the test framework further includes randomly generating test cases as test inputs, performing random mutation according to the variable types obtained in the data extraction and the constraint conditions in the execution path, and using the mutated test inputs as the mutated test inputs, and putting the mutated test inputs back into the split bug-finding test case to obtain newly generated test cases.
6. The automatic patch filtering method based on historical version test oracles according to claim 4, wherein, After obtaining the test oracle, place the newly generated test case in the patch version for execution to obtain the test results in the patch version.
7. An automatic patch filtering system for a method of automatic patch filtering based on historical version test oracles according to any one of claims 1 to 6, characterized in that, It includes a construction module, an extraction module, and a test framework module; the construction module is used to collect the collected code defects and version build datasets; the extraction module is used to extract bug-finding test case data according to the dataset to obtain a test framework for bug-finding test results; the test framework module is used to implement based on the test framework: obtain newly generated test cases, and through the newly generated test cases, execute the newly generated test cases in the historical version and the current version respectively, obtain test oracles and test results, compare the test oracles and the test results, and perform patch filtering verification.