Java virtual machine fuzz testing method based on historical test information feedback

Abstract

The invention discloses a Java virtual machine fuzz test method based on historical test information feedback, which comprises the following steps of: firstly, collecting patch test codes for repairing historical defects from OpenJDK to form a corpus, then analyzing repairing code snippets of the corpus, extracting code snippets related to the defects, and finally, carrying out fuzz test on the defect-related code snippets. And then the defect related code snippets are assembled to obtain a test case, and the defect triggering snippets can be utilized as much as possible to cover deep defect source code branches. And then differential testing is carried out on the assembled test case, feedback scheduling is carried out on defect-related code snippets through historical test information in the differential testing process, and the functional defects and software vulnerabilities of the JVM are found more efficiently.

Description

Technical Field

[0001] The present invention relates to the field of automated software testing, and in particular to a Java virtual machine fuzz testing method based on historical test information feedback. Background Art

[0002] The Java Virtual Machine (JVM) implements an abstract computer platform through software technology. It can not only execute and optimize the Java language, but also is compatible with programs written in other languages ​​such as Scala and Jython. Since JVM has corresponding versions on different operating systems, the design concept of "write once, run everywhere" is realized. The source code of languages ​​such as Java is first compiled into bytecode files, and then loaded and executed by the JVM, which improves the portability and flexibility of the JVM. However, due to the complexity of its design, maintaining and repairing the JVM is not only costly, but also new defects may be introduced during the repair process. Although JVM technology is mature, potential software defects still pose severe challenges. Therefore, it is crucial to ensure the security and correctness of the JVM, and it is particularly critical to accurately identify and repair JVM defects.

[0003] Differential fuzz testing is an automated software defect detection technology that is widely used in the testing of compilers and interpreters. This technology is mainly used to discover problems in the implementation of software functions. However, current research still faces two major challenges, which limit the effectiveness of differential fuzz testing in defect discovery. First, how to construct test cases that can trigger defects is a key issue. Although generating correct test cases is a basic requirement for testing JVM, it is necessary to cover defect-related paths in order to more effectively trigger potential defects in JVM. Secondly, how to use historical information in the testing process to automatically guide the generation of test cases is also a difficulty. In differential testing, the execution result information is often difficult to effectively guide the synthesis of the next round of test cases, so automated feedback to generate test cases that can trigger JVM defects still faces challenges. Summary of the invention

[0004] The object of the present invention is to provide a Java virtual machine fuzz testing method based on historical test information feedback, which can improve the efficiency of JVM fuzz testing.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A Java virtual machine fuzz testing method based on historical test information feedback comprises the following steps:

[0007] Step 1: First, collect patch test codes that repair historical defects from OpenJDK to form a corpus, parse the repair code snippets in the corpus, and extract the defect-related code snippets and the parameter definition statements required to execute the code snippets; secondly, build a seed pool to provide seeds for assembling test cases; finally, use the GCOV tool to instrument the bottom layer of the JVM and record the number of call executions;

[0008] Step 2: Assemble the defect-related code snippets to obtain test cases;

[0009] Step 3: Perform differential testing on the assembled test cases.

[0010] Furthermore, the parsing described in step 1 is performed by using a JavaParser tool.

[0011] Furthermore, the seed pool described in step 1 is constructed by using an open source seed generation method in the COMFUZZ fuzz testing tool to construct the seed pool.

[0012] Furthermore, the assembly process described in step 2 is to comprehensively assemble the defect-related code snippets obtained in step 1, and randomly introduce definition statements that conform to the parameter type to construct a complete test case.

[0013] Furthermore, the assembly process described in step 2 is to combine and splice the defect triggering code snippet in step 1 with the seed in the process of synthesizing the test case.

[0014] Furthermore, the defect triggering code snippet and the seed are combined and spliced ​​to ensure grammatical correctness through the following steps:

[0015] Step 1: When there are undefined variables and unimported package environments in the defect-related code snippets to be assembled, it is necessary to store the missing definitions in the defect-related code snippets into the "context dependency information" list and match them with the API standard information library to generate relevant context codes such as variable declarations and package import statements;

[0016] Step 2: Resolve conflicts between the seed and the defect triggering code snippet;

[0017] Get the existing variable list from the seed, that is, the list of all declared variable names and types from the starting position of the code in the seed to the insertion position of the defect triggering code snippet. By matching with the context dependency information, modify the duplicate variable names, add variable declaration operations to resolve the conflict between the seed and the defect triggering code snippet, and ensure the syntactic correctness of the test case.

[0018] Furthermore, the execution result of the differential test in step 3 includes:

[0019] The execution results of the differential test in step 3 include:

[0020] If the return values ​​of all JVMs are 0 and the output or error information is consistent, it is judged as "passed", that is, the differential test result is "Pass";

[0021] If the return value of all JVMs is 0, but the output information of each JVM is inconsistent, it is judged as "standard output error", that is, the differential test result is "Stdout Error", indicating that there is a problem with the output information of some JVMs;

[0022] Most JVMs return a value of 0 and the execution is successful, while some report an error and return a value of 1, which is considered a "standard error". That is, the differential test result is "Stderr Error", indicating that errors occur during the runtime of some JVMs.

[0023] If the return value of any JVM is -9, the execution result is "timeout", that is, the differential test result is "TimeOut", indicating that the execution time exceeds expectations and there is a performance problem;

[0024] If the return value of any JVM is a negative number other than -9, the execution result is "crash", that is, the differential test result is "Crash", indicating that the JVM has crashed and an error report needs to be submitted.

[0025] Furthermore, the efficiency of differential testing is ensured by the following steps:

[0026] Feedback indicator 1: During the execution of step 3, differential testing usually triggers various abnormal situations. Based on these abnormal feedback, normal test cases with key features collected from the data are added to the seed pool, thereby effectively introducing multiple patch history test fragments;

[0027] Feedback indicator 2: During the execution of step 3, the instrumented JVM can count the coverage of the source code in real time, and extract coverage information through the GCOV tool

[0028] The feedback method is to perform fine-grained coverage quantification on each line or branch of the JVM source code after inserting the above CPP; collect coverage information from the JVM source code in the process of generating test cases from the code segment that triggers the error, and establish the relationship between the code segment that triggers the error and the single line and branch of the JVM source code; detailed coverage information makes it difficult to schedule the defect triggering code segment, and quantify the source code coverage information through information entropy;

[0029] Define the total number of code lines and branches as C, and the coverage frequency of each code line or branch in each CPP file and its header file as f i , according to the coverage frequency f iThe coverage frequency p of each code line or branch i is calculated as the ratio of the total number of source code lines and branches C i , and normalize it using the formula Get the frequency p′ i , where p j is the coverage frequency of each line of source code. Finally, through the formula According to the obtained frequency p′ i Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect-triggering code snippet. Use the information entropy value H(m) to update the priority of the defect-triggering code snippet, and guide the fuzz tester to select the defect-triggering code snippet with a high information entropy value H(m) in subsequent iterations, so as to more effectively discover potential defects.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention first collects patch test codes for repairing historical defects from OpenJDK to form a corpus, then parses the repair code snippets in the corpus, extracts the code snippets related to the defects, and then assembles the defect-related code snippets to obtain test cases, which can cover the deep defect source code branches with the defect triggering snippets as much as possible. Then, differential testing is performed on the assembled test cases to more efficiently discover the functional defects and software vulnerabilities of the JVM.

[0032] Furthermore, the present invention collects test case execution results and coverage information after inserting the JVM source code to form historical test information, and designs two test feedback strategies based on the historical test information, which are further combined with the assembled test cases in step one to reduce the cost of uncovering errors. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is described in detail below in conjunction with specific embodiments.

[0034] The present invention provides a Java virtual machine fuzzy testing method based on historical test information feedback, comprising the following steps:

[0035] Step 1: First, collect patch test codes that repair historical defects from OpenJDK to form a corpus, parse the repair code snippets in the corpus, and extract the defect-related code snippets and the parameter definition statements required to execute the code snippets; secondly, build a seed pool to provide seeds for assembling test cases; finally, use the GCOV tool to instrument the bottom layer of the JVM and record the number of call executions

[0036] The JavaParser tool is used to parse the repair code snippets in the corpus, and the defect-related code snippets are extracted, and the parameter definition statements required for the normal execution of the defect-related code snippets are also extracted; secondly, the seed pool of the present invention is constructed using the open source seed generation method in the COMFUZZ fuzz testing tool to provide seeds for assembling test cases; finally, the GCOV tool is used to instrument the underlying source code of the JVM during the JVM compilation phase, specifically, a counting method is inserted into the CPP file to record the number of call executions, and the assembled test cases are executed using the JVM.

[0037] Step 2: Assemble the defect-related code snippets to obtain test cases

[0038] After fixing historical defects, developers will carefully write corresponding patch test codes. These test cases are specially designed to verify the effectiveness of the fixes. To ensure the security and stability of OpenJDK, developers will update the corresponding test cases synchronously every time the source code is updated. However, regression testing can only check expected situations, while the test cases carefully written by developers usually contain high-quality semantic fragments that can trigger historical defects. Therefore, in order to use these triggering fragments to cover deep-level defective source code branches as much as possible, it is necessary to comprehensively assemble the defect-related code fragments obtained in step one, and randomly introduce definition statements that conform to the parameter type to construct a complete test case. Specifically, in the process of synthesizing test cases, the defect triggering code fragments in step one are combined and spliced ​​with the seeds.

[0039] To ensure the syntactic correctness of the synthetic test cases, the following steps are taken:

[0040] Step 1: When there are undefined variables and unimported package environments in the defect-related code snippets to be assembled, it is necessary to store the missing definitions in the defect-related code snippets into the "context dependency information" list and match them with the API standard information library to generate relevant context codes such as variable declarations and package import statements.

[0041] Step 2: Resolve conflicts between the seed and the bug-triggering code snippet

[0042] There may be conflicts between the seed and the defect triggering code snippet, such as variables with the same name but different types, repeated declarations of the same variable, variables that need to be called but not correctly declared, etc. Get the existing variable list from the seed, that is, the list of all declared variable names and types from the start position of the code in the seed to the insertion position of the defect triggering code snippet, and modify the duplicate variable names by matching with the context dependency information, and add variable declaration operations to resolve the conflict between the seed and the defect triggering code snippet, and ensure the grammatical correctness of the test case.

[0043] In order to fully link the semantics of the seed file and the defect triggering code snippet, the internal position of the seed file is randomly selected during insertion to avoid inserting the start and end positions of the seed file. This not only increases the complexity of the test case, but also provides more complex executable paths.

[0044] Step 3: Perform differential testing on the assembled test cases

[0045] In order to more efficiently discover functional defects and software vulnerabilities of JVM, the present invention adopts an automated differential testing solution to perform fuzz testing on JVM to process test cases. The test involves members from multiple development teams and configures JVMs of various versions of JDK for testing, which can help ensure the compatibility, stability, performance and security of the software in various environments, thereby improving the quality of the software and user experience.

[0046] This paper selects the three most representative open source JVMs: HotSpot, OpenJ9 and GraalVM, and configures two official long-term maintained JDK versions for each JVM: JDK11 and JDK17. A total of six different test environments are prepared for differential testing.

[0047] During the differential test, suspicious test results are determined based on the execution results. The category of the results is mainly determined based on the return value, output information, and error information. When a test case is executed in the differential test, if the return values ​​of all JVMs are 0 and the output or error information is consistent, it is determined to be "passed", that is, the differential test result is "Pass". If the return value of all JVMs is 0, but the output information of each JVM is inconsistent, it is determined to be "standard output error", that is, the differential test result is "Stdout Error", indicating that there is a problem with the output information of some JVMs. If most JVMs return values ​​​​are 0 and the execution passes, and some errors are reported and return values ​​​​are 1, it is determined to be "standard error", that is, the differential test result is "StderrError", indicating that errors occur during the runtime of some JVMs. If the return value of any JVM is -9, the execution result is "timeout", that is, the differential test result is "Time Out", indicating that the execution time exceeds expectations and there are performance problems. If the return value of any JVM is a negative number other than -9, the execution result is "crash", that is, the differential test result is "Crash", indicating that the JVM crashes and an error report needs to be submitted.

[0048] Although the above process can effectively find suspicious results, it takes a lot of time to reveal potential defects by executing a large number of test cases. The present invention designs two feedback indicators to optimize test efficiency.

[0049] Feedback indicator 1. During the execution of fuzz testing, differential testing usually triggers various abnormal situations, such as crashes, errors, and performance degradation. These anomalies are represented by feedback information in different formats. The abnormal information can be extracted through regular matching to accurately determine the type of anomaly in the test case. In addition, analyzing these abnormal behaviors can also locate the code segments that cause defects, and find that different abnormal behaviors have different code coverage characteristics. Based on these abnormal feedbacks, normal test cases with key features collected from the data are added to the seed pool, effectively introducing multiple patch history test fragments. This strategy helps to prioritize the discovery of other important defects in the same module, thereby improving the efficiency and coverage of fuzz testing.

[0050] Feedback indicator 2: During the execution of step 3, the instrumented JVM can count the coverage of the source code in real time, and extract the coverage information through the GCOV tool.

[0051] The JVM interacts with the operating system through the Java native interface, where Java files call CPP files in the JVM source code as native methods, which are responsible for interacting with the operating system and important underlying functions, such as JVM performance optimization.

[0052] In actual calculations, due to the multi-dimensional complexity of coverage information, the feedback method is to perform fine-grained coverage quantification on each line or branch of the JVM source code after instrumenting the above CPP. In order to establish the relationship between the code segment that triggers the error and the single line and branch of the JVM source code, the coverage information after the execution of the test case generated from the code segment that triggers the error is collected from the JVM source code. This includes specific information such as the covered lines and the number of coverage times. Since it is difficult to schedule the defect triggering code segment with detailed coverage information, the source code coverage information is quantified by information entropy.

[0053] Efficient fuzz testing tools can often maximize the amount of information in test cases, build probabilistic measurement models between defective code snippets and sub-modules of the software being tested, and select snippets based on information entropy, which can more efficiently cover branches and trigger new defects.

[0054] Assume that the total number of source code lines and branches is C, and the coverage frequency of each code line or branch i in each CPP file and its header file is f i , according to the coverage frequency f i The coverage frequency p of each code line or branch i is calculated as the ratio of the total number of source code lines and branches C i , and normalize it using the formula Get the frequency p′ i , where p j is the coverage frequency of each line of source code. Finally, through the formula According to the obtained frequency p′ i Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect-triggering code snippet. Use the information entropy value H(m) to update the priority of the defect-triggering code snippet, and guide the fuzz tester to select the defect-triggering code snippet with a high information entropy value H(m) in subsequent iterations, so as to more effectively discover potential defects.

[0055] During the fuzzy testing process, historical test information is formed by collecting test case execution results and collecting coverage information after inserting the JVM source code. The present invention designs two test feedback strategies in combination with the historical test information, and further combines the assembled test cases in step one to reduce the cost of uncovering errors. The above method can effectively reduce the cost of manual analysis, quickly screen a large number of test cases that cannot be uncovered, and more effectively discover JVM defect errors.

[0056] In order to verify the feasibility and effectiveness of the present invention, the present invention carried out the following experiments:

[0057] 1. Experimental environment

[0058] During the implementation of this solution, the experimental platform was built on a high-performance server with an Intel(R) Core(TM) i9-9940X processor with a main frequency of 3.30GHz and an Ubuntu 18.04 (kernel version 5.4.0) operating system, 64GB of memory, and two RTX 2080Ti graphics cards. The test case seed selected the open source dataset of COMFUZZ fuzz testing.

[0059] 2. Experimental process

[0060] (1) The extracted defect-related code snippets are as follows:

[0061] if((slotID==-1)&&(slotListIndex==-1)){

[0062] return 0;

[0063] }else{

[0064] return slotListIndex;

[0065] }

[0066] (2) The test cases that were successfully assembled are as follows:

[0067]

[0068]

[0069] In the test case synthesis process, we first select seeds from the seed pool and assemble them with the code that triggers the error, and then introduce the unsafe definition statement. The test case containing unsafe is the code snippet that triggered the exception in the previous round of testing and was added to the seed pool. In this synthesis, the synthesis process selected the for code block based on the priority of the code block. When inserting, it is necessary to introduce undefined variables addr, len, and value based on context analysis. Finally, the undefined variable ByteBuffer b was found when scanning the context dependency, and it was introduced through the API standard information library matching database. After several rounds of assembly, we obtained a syntactically correct test case.

[0070] 3. Experimental results

[0071] (1) Performance defect discovery

[0072] By testing 50,000 generated test cases on three major mainstream JVMs for 24 hours, we obtained a statistical table of discovered bugs, as shown in Table 1. So far, a total of 6 different performance defects have been found, 6 of which have been confirmed by engine developers, and 3 defects have been fixed by developers.

[0073] Table 1 JVM defects found

[0074] Engine Name Number of submissions Accept quantity Fixed HotSpot 2 2 1 OpenJ9 3 3 2 GraalVM 1 1 0 total 6 6 3

[0075] (2) Examples of Defects Found

[0076] The test case that triggers the functional error is as follows:

[0077]

[0078]

[0079] In the test case that triggers the functional error above, select the code block with higher priority from lines 10 to 13 to synthesize a new test case, namely:

[0080]

[0081] Since optionCount and rm are undefined variables in the code block, and rm has been initialized in line 6, we only need to add the optionCount variable in line 7, introduce the buf variable, and successfully match it with the buf in the seed in line 2 to finally synthesize the test case.

[0082] The logic of triggering the API bug in OpenJ9 is as follows: The seed file of this test case comes from the iteration result of the last abnormal feedback. After manual analysis, the system judged that the call to the ManagementFactory.getPlatformMXBeans function on line 11 was abnormal when obtaining ThreadIds, but because it met the false positive rule, a test case without errors was put into the seed pool through abnormal behavior feedback. Combined with the new code segment that triggers the defect on lines 10-13, when calling clear, due to the design of getInputArg nts in OpenJ9, the return value is designed to be a modifiable list, resulting in a memory thread leak, triggering the interface defect of OpenJ9. Finally, the developer fixed the defect.

Claims

1. A Java virtual machine fuzz testing method based on historical test information feedback, characterized in that: The following steps are involved: Step 1: First, collect patch test codes that fix historical defects from OpenJDK to form a corpus, parse the repair code snippets in the corpus, and extract the defect-related code snippets and the parameter definition statements required to execute the code snippets; Secondly, we build a seed pool to provide seeds for assembling test cases. Finally, we use the GCOV tool to instrument the underlying JVM and record the number of execution calls. Step 2: Assemble the defect-related code snippets to obtain test cases; Step 3: Perform differential testing on the assembled test cases.

2. A Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that: The parsing described in step 1 is performed by using the JavaParser tool.

3. A Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that: The seed pool described in step 1 is constructed by using the open source seed generation method in the COMFUZZ fuzz testing tool.

4. A Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that: The assembly process described in step 2 is to comprehensively assemble the defect-related code snippets obtained in step 1, and randomly introduce definition statements that conform to the parameter type to construct a complete test case.

5. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1 is characterized in that: The assembly process described in step 2 is to combine and splice the defect triggering code snippet in step 1 with the seed in the process of synthesizing the test case.

6. A Java virtual machine fuzz testing method based on historical test information feedback according to claim 5, characterized in that: The defect triggering code snippet and the seed are combined and spliced ​​to ensure grammatical correctness through the following steps: Step 1: When there are undefined variables and unimported package environments in the defect-related code snippets to be assembled, it is necessary to store the missing definitions in the defect-related code snippets into the "context dependency information" list and match them with the API standard information library to generate relevant context codes such as variable declarations and package import statements; Step 2: Resolve conflicts between the seed and the defect triggering code snippet; Get the existing variable list from the seed, that is, the list of all declared variable names and types from the starting position of the code in the seed to the insertion position of the defect triggering code snippet. By matching with the context dependency information, modify the duplicate variable names, add variable declaration operations to resolve the conflict between the seed and the defect triggering code snippet, and ensure the syntactic correctness of the test case.

7. A Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that: The execution results of the differential test in step 3 include: If the return values ​​of all JVMs are 0 and the output or error information is consistent, it is judged as "passed", that is, the differential test result is "Pass"; If the return value of all JVMs is 0, but the output information of each JVM is inconsistent, it is judged as "Standard Output Error", that is, the differential test result is "Stdout Error", indicating that there is a problem with the output information of some JVMs; Most JVMs return a value of 0 and the execution is successful, while some report an error and return a value of 1, which is considered a "standard error". That is, the differential test result is "Stderr Error", indicating that errors occur during the runtime of some JVMs. If the return value of any JVM is -9, the execution result is "Timeout", that is, the differential test result is "Time Out", indicating that the execution time exceeds expectations and there is a performance problem; If the return value of any JVM is a negative number other than -9, the execution result is "crash", that is, the differential test result is "Crash", indicating that the JVM has crashed and an error report needs to be submitted.

8. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that: To ensure the efficiency of differential testing, follow these steps: Feedback indicator 1: During the execution of step 3, differential testing usually triggers various abnormal situations. Based on these abnormal feedback, normal test cases with key features collected from the data are added to the seed pool, thereby effectively introducing multiple patch history test fragments; Feedback indicator 2: During the execution of step 3, the instrumented JVM can count the coverage of the source code in real time, and extract coverage information through the GCOV tool The feedback method is to perform fine-grained coverage quantification on each line or branch of the JVM source code after instrumenting the above CPP; collect coverage information from the JVM source code in the process of generating test cases from the code segment that triggers the error, and establish the relationship between the code segment that triggers the error and the single line and branch of the JVM source code; Detailed coverage information makes it difficult to schedule defect-triggering code snippets, and source code coverage information is quantified through information entropy; Define the total number of code lines and branches as C, and the coverage frequency of each code line or branch in each CPP file and its header file as f i , according to the coverage frequency f i The coverage frequency p of each code line or branch i is calculated as the ratio of the total number of source code lines and branches C i , and normalize it using the formula Get the frequency p i ′, where p j is the coverage frequency of each line of source code. Finally, through the formula According to the obtained frequency p i 'Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect triggering code fragment. Use the information entropy value H(m) to update the priority of the defect triggering code fragment, and guide the fuzz tester to select the defect triggering code fragment with a high information entropy value H(m) in subsequent iterations, so as to more effectively discover potential defects.

Images