Java virtual machine fuzzing method based on historical test information feedback

Abstract

The application discloses a Java virtual machine fuzzing test method based on historical test information feedback, and first, a corpus is formed by collecting patch test codes for repairing historical defects from OpenJDK, then the repair code segments of the corpus are parsed, and defect-related code segments are extracted, and the defect-related code segments are assembled to obtain test cases, so that defect triggering segments can be used as much as possible to cover deep defect source code branches. Then, the assembled test cases are subjected to differential testing, and the defect-related code segments are fed back and scheduled through historical test information in the differential testing process, so that functional defects and software vulnerabilities of the JVM can be found more efficiently.

Description

Technical Field

[0001] This invention relates to the field of automated software testing, and more specifically to a Java Virtual Machine fuzzing method based on historical test information feedback. Background Technology

[0002] The Java Virtual Machine (JVM) implements an abstract computer platform through software technology. It not only executes and optimizes programs written in Java but is also compatible with programs written in other languages ​​such as Scala and Jython. Because JVM versions exist for different operating systems, it embodies the "write once, run anywhere" design philosophy. Source code for languages ​​like Java is first compiled into bytecode files, which are then loaded and executed by the JVM, improving its portability and flexibility. However, due to its complex design, maintaining and fixing the JVM is not only costly but may also introduce new defects during the fixing process. Despite the maturity of JVM technology, potential software defects still pose a significant challenge. Therefore, ensuring the security and correctness of the JVM is crucial, and accurately identifying and fixing JVM defects is particularly critical.

[0003] Differential fuzzing is an automated software defect detection technique widely used in compiler and interpreter testing. This technique is primarily used to discover problems in software functionality. However, current research still faces two main challenges that limit the effectiveness of differential fuzzing in defect discovery. First, how to construct test cases that can trigger defects is a key issue. While generating correct test cases is a fundamental requirement for testing the JVM, they must cover defect-related paths to more effectively trigger potential JVM defects. Second, how to automatically guide test case generation using historical information from the testing process is also a challenge. In differential testing, execution result information is often insufficient to effectively guide the synthesis of test cases in the next round; therefore, automatically generating test cases that can trigger JVM defects remains a challenge. Summary of the Invention

[0004] The purpose of this invention is to provide a Java Virtual Machine fuzzing method based on historical test information feedback that can improve the efficiency of JVM fuzzing.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A Java Virtual Machine fuzzing method based on historical test information feedback includes the following steps:

[0007] Step 1: First, collect patch test code that fixes historical defects from OpenJDK to form a corpus. Parse the patch code snippets in the corpus and extract the defect-related code snippets and the parameter definition statements required to execute the code snippets. Second, build a seed pool to provide seeds for assembling test cases. Finally, use the GCOV tool to instrument the JVM's underlying layer and record the number of execution calls.

[0008] Step 2: Assemble the defect-related code snippets to obtain test cases;

[0009] Step 3: Perform differential testing on the assembled test cases.

[0010] Furthermore, the parsing described in step one is performed using the JavaParser tool.

[0011] Furthermore, the seed pool described in step one is constructed using the open-source seed generation method in the COMFUZZ fuzzing tool.

[0012] Furthermore, the assembly process described in step two involves comprehensively assembling the defect-related code fragments obtained in step one, and randomly introducing definition statements that conform to the parameter types to construct complete test cases.

[0013] Furthermore, the assembly process described in step two involves combining and splicing the defect triggering code fragment from step one with the seed during the synthesis of test cases.

[0014] Furthermore, the defect-triggered code snippet and the seed are combined and concatenated using the following steps to ensure syntactic correctness:

[0015] Step 1: When there are undefined variables or missing package environments in the defect-related code snippets to be assembled, the missing definitions in the defect-related code snippets need to be stored in the "context dependency information" list and matched with the API standard information library to generate relevant context code such as variable declarations and package import statements;

[0016] Step 2: Resolve the conflict between the seed and the defect triggering code snippet;

[0017] The test case retrieves a list of existing variables from the seed, which is a list of all declared variable names and types from the start of the code in the seed to the insertion point of the defect-triggered code snippet. By matching these with context dependency information, duplicate variable names are modified, and variable declarations are added to resolve conflicts between the seed and the defect-triggered code snippet, ensuring the syntactic correctness of the test case.

[0018] Furthermore, the execution results of the differential test described in step three include:

[0019] The results of the differential test described in step three include:

[0020] If all JVM return values ​​are 0 and the output or error message is consistent, it is considered "passed", that is, the differential test result is "Pass".

[0021] If all JVMs return 0, but the output information of each JVM is inconsistent, it is judged as "Stdout Error", that is, the differential test result is "Stdout Error", indicating that there is a problem with the output information of some JVMs;

[0022] Most JVMs return a value of 0 and the execution passes, while some return a value of 1 while reporting an error. This is considered a "standard error", meaning the differential test result is "Stderr Error", indicating that an error occurred during the runtime of some JVMs.

[0023] If any JVM returns -9, the execution result is "Timeout", which means the differential test result is "TimeOut", indicating that the execution time exceeds the expectation and there is a performance problem.

[0024] If any JVM returns a negative number other than -9, the execution result is "crash," meaning the differential test result is "Crash," indicating that the JVM has crashed and an error report needs to be submitted.

[0025] Furthermore, the efficiency of differential testing is ensured through the following steps:

[0026] Feedback Metric 1: During the execution of step 3, differential testing usually triggers various abnormal situations. Based on these abnormal feedbacks, normal test cases with key characteristics collected from the data are added to the seed pool, thereby effectively introducing multiple patch history test segments.

[0027] Feedback Metric 2: During the execution of step 3, the instrumented JVM can perform real-time statistics on source code coverage and extract coverage information using the GCOV tool.

[0028] The feedback method involves instrumenting the aforementioned CPP and then performing fine-grained coverage quantification on each line or branch of the JVM source code; collecting coverage information from the JVM source code during the process of generating test cases from the code segment that triggers the error, and establishing the relationship between the code segment that triggers the error and the single line and branch of the JVM source code; detailed coverage information makes it difficult to schedule defect-triggered code segments, so source code coverage information is quantified through information entropy.

[0029] Let C be the total number of lines of code and branches, and f be the frequency of coverage for each line of code or branch in each CPP file and its header file. i According to the coverage frequency f iThe coverage frequency p of each line of code or branch i is calculated as the ratio of the total number of source code lines and branches, which is C. i And perform normalization using the formula The frequency p′ is obtained i In the formula p j To determine the coverage frequency of each line of source code, finally, using the formula... Based on the obtained frequency p′ i Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect triggering code segment. Use the information entropy value H(m) to update the priority of the defect triggering code segment, and guide the fuzz tester to select the defect triggering code segment with a high information entropy value H(m) in subsequent iterations, so as to discover potential defects more effectively.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention first constructs a corpus by collecting patch test code from OpenJDK to fix historical defects. Then, it parses the patch code snippets in the corpus to extract defect-related code snippets, and assembles these snippets to obtain test cases. This approach maximizes the use of defect-triggered snippets to cover deep defective source code branches. Finally, differential testing is performed on the assembled test cases to more efficiently discover JVM functional defects and software vulnerabilities.

[0032] Furthermore, this invention collects test case execution results and coverage information after instrumenting JVM source code to form historical test information. Based on this historical test information, two test feedback strategies are designed, which, combined with the assembled test cases in step one, reduce the cost of debugging. Detailed Implementation

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be described in detail below with reference to specific embodiments.

[0034] This invention provides a Java Virtual Machine fuzzing method based on historical test information feedback, comprising the following steps:

[0035] Step 1: First, collect patch test code from OpenJDK to fix historical defects and form a corpus. Parse the patch code snippets in the corpus and extract the defect-related code snippets and the parameter definition statements required to execute them. Second, build a seed pool to provide seeds for assembling test cases. Finally, use the GCOV tool to instrument the JVM's underlying layers and record the number of execution calls.

[0036] The JavaParser tool is used to parse the repair code snippets in the corpus, extracting the code snippets related to the defects, and also extracting the parameter definition statements required for the normal execution of the defect-related code snippets. Secondly, the seed pool of this invention is constructed using the open-source seed generation method in the COMFUZZ fuzzing tool to provide seeds for assembling test cases. Finally, during the JVM compilation stage, the GCOV tool is used to instrument the underlying source code of the JVM, specifically inserting counting methods into the CPP file to record the number of times the methods are called, and then using the JVM to execute the assembled test cases.

[0037] Step 2: Assemble the defect-related code snippets to obtain test cases.

[0038] After fixing historical defects, developers carefully write corresponding patch test code. These test cases are specifically designed to verify the effectiveness of the fix. To ensure the security and stability of OpenJDK, developers update the corresponding test cases synchronously with each source code update. However, regression testing can only check expected scenarios, while carefully written test cases typically contain high-quality semantic fragments that can trigger historical defects. Therefore, to utilize these triggering fragments to cover deeper defective source code branches as much as possible, it is necessary to synthesize and assemble the defect-related code fragments obtained in step one, and randomly introduce definition statements that conform to parameter types to construct complete test cases. Specifically, in the process of synthesizing test cases, the defect-triggered code fragments from step one are combined and spliced ​​with the seed.

[0039] To ensure the syntactic correctness of synthetic test cases, the following steps are taken:

[0040] Step 1: When undefined variables or missing package environments exist in the defect-related code snippets to be assembled, the missing definitions in the defect-related code snippets need to be stored in the "Context Dependency Information" list and matched with the API standard information library to generate relevant context code such as variable declarations and package import statements.

[0041] Step 2: Resolve the conflict between the seed and the defect triggering code snippet

[0042] Conflicts may exist between the seed and the defect-triggered code snippet, such as variables with the same name but different types, duplicate declarations of the same variable, or incorrectly declared variables that need to be called. The test case retrieves a list of existing variables from the seed, that is, a list of all declared variable names and types from the start of the code in the seed to the insertion point of the defect-triggered code snippet. By matching this list with context dependency information, duplicate variable names are modified, and variable declarations are added to resolve conflicts between the seed and the defect-triggered code snippet, ensuring the syntactic correctness of the test case.

[0043] To fully link the semantics of the seed file and the defect-triggered code snippets, the position inside the seed file is randomly selected during insertion, avoiding insertion at the beginning and end positions of the seed file. This increases the complexity of the test cases and provides more complex executable paths.

[0044] Step 3: Perform differential testing on the assembled test cases.

[0045] To more efficiently discover JVM functional defects and software vulnerabilities, this invention employs an automated differential testing scheme to perform fuzz testing on the JVM to process test cases. The testing involves members from multiple development teams and uses JVMs with various JDK versions for testing. This helps ensure the software's compatibility, stability, performance, and security in various environments, thereby improving software quality and user experience.

[0046] This invention selected three of the most representative open-source JVMs: HotSpot, OpenJ9, and GraalVM, and configured two officially maintained long-term JDK versions for each JVM: JDK11 and JDK17, preparing a total of six different test environments for differential testing.

[0047] During differential testing, suspicious test results are determined based on the execution results, primarily relying on return values, output information, and error messages to classify the results. When a test case is executed in differential testing, if all JVMs return values ​​of 0 and their output or error messages are consistent, it is considered a "pass," meaning the differential test result is "Pass." If all JVMs return values ​​of 0, but their output information is inconsistent, it is considered a "Stdout Error," indicating a problem with the output information of some JVMs. If most JVMs return values ​​of 0 and execution passes, while some return values ​​of 1 and error messages, it is considered a "Stderr Error," indicating a runtime error in some JVMs. If any JVM returns a value of -9, the execution result is a "Timeout," meaning the execution time exceeds expectations and there is a performance issue. If any JVM returns a negative number other than -9, the execution result is a "Crash," indicating a JVM crash, requiring an error report to be submitted.

[0048] While the above process can effectively identify suspicious results, revealing potential defects by executing a large number of test cases requires a significant amount of time. This invention designs two feedback metrics to optimize testing efficiency.

[0049] The first feedback metric is that during fuzzing, differential testing typically triggers various anomalies, such as crashes, errors, and performance degradation. These anomalies are represented by feedback messages in different formats. Anomaly information can be extracted using regular expression matching, thus accurately determining the anomaly type in the test cases. Furthermore, analyzing these anomaly behaviors can pinpoint the code segments causing defects and reveal that different anomaly behaviors have different code coverage characteristics. Based on these anomaly feedbacks, normal test cases with key characteristics collected from the data are added to the seed pool, effectively introducing multiple patch history test segments. This strategy helps to prioritize the discovery of other important defects in the same module, thereby improving the efficiency and coverage of fuzzing.

[0050] Feedback metric two: During the execution of step three, the instrumented JVM can statistically analyze the source code coverage in real time and extract coverage information through the GCOV tool.

[0051] The JVM interacts with the operating system through the Java Native Interface. Java files call CPP files in the JVM source code as native methods, which are responsible for interacting with the operating system and important low-level functions, such as JVM performance optimization.

[0052] In practical calculations, due to the multidimensional complexity of coverage information, this feedback method involves instrumenting the aforementioned CPP (Code Segment) and then performing fine-grained coverage quantification on each line or branch of the JVM source code. To establish the relationship between the error-triggered code segment and individual lines and branches of the JVM source code, coverage information after executing test cases generated from the error-triggered code segment is collected from the JVM source code. This includes specific information such as the covered lines and the number of coverage iterations. Since detailed coverage information is difficult to schedule defect-triggered code segments, source code coverage information is quantified using information entropy.

[0053] Efficient fuzzing tools can maximize the amount of information in test cases, build a probability metric model between defective code snippets and sub-modules of the software under test, and select snippets based on information entropy, which can more efficiently cover branches and trigger new defects.

[0054] Assume the total number of source code lines and branches is C, and the frequency of coverage for each line or branch i in each CPP file and its header file is f. i According to the coverage frequency f i The coverage frequency p of each line of code or branch i is calculated as the ratio of the total number of source code lines and branches, which is C. i And perform normalization using the formula The frequency p′ is obtained i In the formula p j This represents the coverage frequency of each line of source code. Finally, it is calculated using the formula... Based on the obtained frequency p′ i Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect triggering code segment. Use the information entropy value H(m) to update the priority of the defect triggering code segment, and guide the fuzz tester to select the defect triggering code segment with a high information entropy value H(m) in subsequent iterations, so as to discover potential defects more effectively.

[0055] During fuzzing, historical test information is compiled by collecting test case execution results and coverage information after instrumenting the JVM source code. This invention incorporates this historical test information to design two test feedback strategies, further combining them with the test case assembly in step one to reduce the cost of bug detection. This method effectively reduces manual analysis costs, quickly filters out a large number of test cases that cannot be detected, and more effectively discovers JVM defects and errors.

[0056] To verify the feasibility and effectiveness of this invention, the following experiments were conducted:

[0057] 1. Experimental Environment

[0058] During the implementation of this solution, the experimental platform was built on a high-performance server with an Intel(R) Core(TM) i9-9940X processor with a main frequency of 3.30GHz and an Ubuntu 18.04 (kernel version 5.4.0) operating system, 64GB of memory, and two RTX 2080Ti graphics cards. The test case seeds were selected from the open-source dataset of COMFUZZ fuzzing test.

[0059] 2. Experimental Procedure

[0060] (1) The extracted defect-related code snippets are as follows:

[0061] if((slotID==-1)&&(slotListIndex==-1)){

[0062] return 0;

[0063] }else{

[0064] return slotListIndex;

[0065] }

[0066] (2) The test cases that were successfully assembled are as follows:

[0067]

[0068]

[0069] During test case synthesis, a seed is first selected from the seed pool and assembled with the code that triggered the error. Then, an unsafe definition statement is introduced. Test cases containing unsafe statements are code snippets that triggered exceptions in the previous test and were added to the seed pool. In this synthesis, a for loop was selected based on code block priority. During insertion, undefined variables addr, len, and value need to be introduced based on context analysis. Finally, an undefined variable ByteBuffer b was found during context dependency scanning and introduced by matching it to the database using the API standard information repository. After several rounds of assembly, we obtained syntactically correct test cases.

[0070] 3. Experimental Results

[0071] (1) Performance defect discovery status

[0072] 50,000 generated test cases were tested for 24 hours on three major JVMs. The resulting bug statistics are shown in Table 1. To date, six different performance defects have been discovered, all of which have been confirmed by the engine developers, and three have been fixed.

[0073] Table 1 shows the JVM defects found.

[0074] Engine Name Number of submissions Number of accepted Fixed HotSpot 2 2 1 OpenJ9 3 3 2 GraalVM 1 1 0 total 6 6 3

[0075] (2) Examples of discovered defects

[0076] The test cases that trigger functional errors are shown below:

[0077]

[0078]

[0079] In the test cases that trigger functional errors mentioned above, the code block with higher priority from lines 10 to 13 is selected and combined into a new test case, namely:

[0080]

[0081] Since optionCount and rm are undefined variables in the code block, and rm has been initialized in line 6, we only need to add the optionCount variable in line 7, introduce the buf variable, and successfully match it with buf in the seed in line 2 to finally synthesize the test case.

[0082] The logic that triggered the API bug in OpenJ9 is as follows: The seed file for this test case comes from the iteration result of the previous exception feedback. After manual analysis, it was found that the function `ManagementFactory.getPlatformMXBeans` on line 11 was judged as an exception by the system when retrieving `ThreadIds`. However, because it met the false positive rule, a test case without errors was added to the seed pool through exception behavior feedback. Combined with the new code segment that triggered the defect on lines 10-13, when `clear` was called, due to the design of `getInputArg nts` in OpenJ9, the return value was designed as a modifiable list, which caused a memory thread leak, triggering the OpenJ9 interface defect. Ultimately, the developers fixed the defect.

Claims

1. A Java virtual machine fuzz testing method based on historical test information feedback, characterized in that, The method comprises the following steps: Step one: first, collect the patch test code for repairing historical defects from OpenJDK to form a corpus, parse the repair code fragments in the corpus, and extract the defect-related code fragments and the parameter definition statements required for executing the code fragments; Secondly, build a seed pool to provide seeds for assembling test cases; finally, insert a probe into the bottom layer of JVM through the GCOV tool to record the number of calls; Step two: assemble the defect-related code fragments to obtain test cases; Step three: perform differential testing on the assembled test cases; The efficiency of differential testing is ensured through the following steps: Feedback index one: during the execution of step three, differential testing usually triggers various abnormal situations, and according to these abnormal feedbacks, normal test cases with key features collected from the data are added to the seed pool, thereby effectively introducing multiple patch history test fragments; Feedback index two: during the execution of step three, the patched JVM real-time statistics the coverage of the source code, and the GCOV tool is used to extract the coverage information; The feedback method is to perform fine-grained coverage quantization on each line or each branch of the JVM source code after inserting a probe into the CPP; Collect the coverage information in the process of generating test cases from the error-triggering code fragments from the JVM source code, and establish the relationship between the error-triggering code fragments and the single line and branch of the JVM source code; The detailed coverage information is difficult to schedule the defect-triggering code fragments, and the source code coverage information is quantized through information entropy. Define the total number of lines of code and branches as The frequency of coverage for each line of code or branch in each CPP file and its header file is: According to coverage frequency The total number of source code lines and branches is The ratio is calculated for each line of code or branch. Coverage frequency And perform normalization using the formula Obtain frequency In the formula To determine the coverage frequency of each line of source code, finally, using the formula... Based on the obtained frequency Calculate the information entropy value H(m) of the current source code coverage. The higher the information entropy value H(m), the higher the priority of the defect triggering code segment. Use the information entropy value H(m) to update the priority of the defect triggering code segment, and guide the fuzz tester to select the defect triggering code segment with a high information entropy value H(m) in subsequent iterations, so as to discover potential defects more effectively.

2. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that, The parsing in step one is performed by using the JavaParser tool.

3. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that, The construction of the seed pool in step one is to use the open source seed generation method in the COMFUZZ fuzz testing tool to construct the seed pool.

4. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that, The assembly process in step two is to comprehensively assemble the defect-related code fragments obtained in step one, and randomly introduce definition statements conforming to the parameter types to construct complete test cases.

5. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that, The assembly process in step two is to combine and splice the defect-triggering code fragments in step one with the seeds during the synthesis of test cases.

6. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 5, characterized in that, The defect-triggering code fragments and the seeds are combined and spliced through the following steps to ensure syntax correctness: Step 1: If there are undefined variables and no package environment in the defect-related code fragments to be assembled, the missing definitions in the defect-related code fragments are stored in the "context dependency information" list, and the variable declaration, package import statement related context code is generated by matching with the API standard information library; Step 2: Solve the conflict between the seed and the defect-triggering code fragments; Get the existing variable list from the seed, that is, the list of all declared variable names and types from the starting position of the seed code to the insertion position of the defect-triggering code fragments, modify the repeated variable names, and add variable declaration operations to solve the conflict between the seed and the defect-triggering code fragments, and ensure the syntax correctness of the test cases.

7. The Java virtual machine fuzz testing method based on historical test information feedback according to claim 1, characterized in that, The execution results of the differential testing in step three include: All JVMs return 0 and the output or error information is consistent, then it is determined as "Pass", i.e. the differential test result is "Pass"; All JVMs return 0, but the output information of each JVM is inconsistent, then it is determined as "Standard Output Error", i.e. the differential test result is "Stdout Error", indicating that the output information of some JVMs has a problem; Most JVMs return 0 and pass, while some return 1 and report an error, then it is determined as "Standard Error", i.e. the differential test result is "Stderr Error", indicating that some JVMs have an error during execution; The return value of any JVM is -9, then the execution result is "Time Out", i.e. the differential test result is "Time Out", indicating that the execution time exceeds the expectation and there is a performance problem; The return value of any JVM is a negative number other than -9, then the execution result is "Crash", i.e. the differential test result is "Crash", indicating that the JVM crashes and an error report needs to be submitted.

Images