A vulnerability mining method of multi-static analysis tool result fusion

Abstract

This invention provides a vulnerability mining method based on the fusion of results from multiple static analysis tools, comprising: Step 1, scanning a benchmark dataset using multiple static analysis tools and calculating the confidence level of each tool based on the results; Step 2, performing function-level slicing on the project under test and extracting function code attribute graphs, and representing the code attribute graphs using a code attribute graph specification dictionary to obtain code attribute graph vectors; Step 3, further detecting the project under test and normalizing and fusing the detection reports from each static analysis tool; Step 4, representing the normalized and fused detection report as a feature vector by combining the code attribute graph vectors; Step 5, inputting the feature vectors into a pre-trained classification model. If the classification model determines the detection report as a positive report, it indicates that a vulnerability exists at the corresponding location. This invention can alleviate the severe false positive rate of static analysis methods, effectively reducing false positives by 90% when tested on a benchmark test set.

Description

Technical Field

[0001] This invention relates to the field of vulnerability detection, and in particular to a vulnerability discovery method that integrates the results of multiple static analysis tools. Background Technology

[0002] Software vulnerabilities refer to defects that arise in software throughout its lifecycle (i.e., the entire process of design, development, deployment, and execution). These defects can be exploited by hackers to bypass system access controls, illegally gain access, leak user privacy data, steal digital assets, and cause huge economic losses to businesses and individuals. According to data published by the U.S. National Vulnerability Database (NVD), as of 2022, more than 180,000 code vulnerabilities had been disclosed. The Apache log4j CVE-2021-44228 vulnerability alone affected more than 35,000 Java packages, impacting one-third of servers worldwide, and more than 800,000 exploit attempts were detected within 72 hours of its disclosure. Therefore, timely detection of vulnerabilities in software is crucial.

[0003] Currently, software vulnerability detection methods are mainly divided into two categories: static analysis and dynamic testing. Static analysis refers to scanning the source code without running the code, using methods such as lexical analysis, syntax analysis, and rule matching to uncover potential defects. Dynamic testing refers to software vulnerability discovery methods that provide unexpected input to the running program under test and monitor abnormal output results. Each method has its advantages and disadvantages. Static analysis is fast and has good code coverage, but it has a higher false positive rate. Dynamic testing has a low false positive rate and high accuracy, but it is very time-consuming.

[0004] As an important source code vulnerability detection technique, various static analysis tools have been developed by the research and industry communities both domestically and internationally. Each tool has its own technical mechanism and underlying implementation. For example, FlawFinder and CppCheck directly perform rule matching based on the source code, outputting an alert as long as a preset defect pattern is matched; while CodeQL and ShiftLeft transform the source code into a more abstract intermediate representation and then perform detection based on this intermediate representation. This results in each tool being able to detect different types of code defects and having different use cases, and there is no single tool that can be considered a universal solution. Summary of the Invention

[0005] To address the problems existing in the current technology, a vulnerability mining method based on the fusion of results from multiple static analysis tools is provided. This method conducts an in-depth study of the advantages, disadvantages, performance, and confidence level of each static tool for specific defects. By combining machine learning techniques to weigh the tool results, the method aggregates the high-confidence detection results from multiple tools, discards low-confidence defect alerts, and optimizes the complex fusion problem, thereby alleviating the serious false alarm rate of static analysis methods.

[0006] The technical solution adopted in this invention is as follows: A vulnerability mining method based on the fusion of results from multiple static analysis tools, comprising:

[0007] Step 1: Slice the project under test into function-level slices and extract the function code attribute graphs. Combine the code attribute graph specification dictionary to represent the code attribute graphs and obtain code attribute graph vectors.

[0008] Step 2: Scan the benchmark dataset using multiple static analysis tools and calculate the confidence level of each static analysis tool based on the results;

[0009] Step 3: Then, use multiple static analysis tools to test the project under test, and normalize and merge the test reports from each static analysis tool.

[0010] Step 4: Represent the normalized and fused detection report as a feature vector, wherein the feature vector consists of a code attribute graph vector and the confidence scores of each static analysis tool;

[0011] Step 5: Input the feature vector into the pre-trained classification model. If the classification model determines that the detection report is a positive report, it means that there is a vulnerability at the corresponding location.

[0012] Furthermore, in step 1, the benchmark dataset is Juliet-test-suite.

[0013] Furthermore, in step 2, using the benchmark dataset and the confidence assessment system, the F1 score is calculated based on the line-of-code granularity for the analysis results output of the static analysis tool, and the F1 score is used as the confidence score of the corresponding static analysis tool.

[0014] Furthermore, the confidence assessment system is specifically as follows:

[0015] Positive TP report: The test case line x contains defect i, and the tool reports that line x contains defect i;

[0016] False positive (FP): There is no defect in line x of the test case, but the tool reports that there is defect i in line x;

[0017] FN Miss: The test case has defect i in line x, but the tool did not report it;

[0018] True negative TN: There is no defect in line x of the test case, and the tool does not report it;

[0019] Precision rate:

[0020] Recall rate:

[0021] F1 value:

[0022] Furthermore, in step 2, the code attribute graph standard dictionary consists of 140 corpus elements. The vector representation process requires parsing and traversing the code attribute graph, placing the nodes and edges in the code attribute graph into a list sequence in sequence, and finally comparing the list with the dictionary to calculate the frequency of dictionary elements appearing in the list sequence.

[0023] Furthermore, in step 1, the code attribute graph vector is represented as:

[0024]

[0025] in, g is a dictionary for code property graph specification. z = (v, e), where v represents the set of vertices and e represents the set of edges.

[0026] Furthermore, in step 3, during the normalization fusion process, the analysis results with low confidence are discarded based on the confidence level of the static analysis tool.

[0027] Furthermore, in step 4, the feature vector is represented as:

[0028]

[0029] in, Let z be the code attribute graph vector, m*w ta m*w tb m*w tc m*w td This indicates whether static analysis tools a, b, c, and d have this test report. The value of m is 0 or 1, where 0 indicates no test report and 1 indicates a test report exists. ta w tb w tc w td These represent the confidence levels of the defects y detected by static analysis tools a, b, c, and d, respectively.

[0030] Furthermore, the classification model training process is as follows: the benchmark dataset is divided into a training set and a test set according to the proportion; the training set is scanned using a static analysis tool to obtain a detection report; the report is converted into a feature vector using a report fusion representation method; and the fused feature set is labeled in combination with the benchmark data to supervise the training of the model.

[0031] Compared with the existing technology, the beneficial effects of adopting the above technical solution are as follows: the present invention aggregates the high-reliability detection results of multiple tools, discards low-reliability defect alarms, optimizes the complex fusion problem, can alleviate the serious false alarm rate of static analysis methods, and effectively reduces false alarms by 90% when tested on a benchmark test set. Attached Figure Description

[0032] Figure 1 This is a flowchart of the vulnerability mining method based on the fusion of results from multiple static analysis tools proposed in this invention.

[0033] Figure 2 This is a schematic diagram of feature vectors in one embodiment of the present invention. Detailed Implementation

[0034] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0035] like Figure 1 As shown, this embodiment proposes a vulnerability mining method that fuses the results of multiple static analysis tools. It primarily addresses the problems and limitations of existing static analysis techniques, solving the issues of different applicable scenarios, varying defect coverage categories, and high false positive rates among static analysis tools. The outputs of multiple static analysis tools are normalized, and the results are represented as feature vectors that fuse function code attributes and tool confidence. These vectors are then input into a pre-trained machine learning classification model to predict whether the detection result is true. The specific scheme is as follows:

[0036] Step 1: Scan the benchmark dataset using multiple static analysis tools and calculate the confidence level of each static analysis tool based on the results;

[0037] Step 2: Slice the project under test into function-level slices and extract the function code attribute graphs. Combine the code attribute graph specification dictionary to represent the code attribute graphs and obtain code attribute graph vectors.

[0038] Step 3: Then, use multiple static analysis tools to test the project under test, and normalize and merge the test reports from each static analysis tool.

[0039] Step 4: Combine the code attribute graph vector to represent the normalized and fused detection report as a feature vector;

[0040] Step 5: Input the feature vector into the pre-trained classification model. If the classification model determines that the detection report is a positive report, it means that there is a vulnerability at the corresponding location.

[0041] In this embodiment, to utilize existing detection tools for fusion, a thorough understanding of the tools is first required. Understanding the detection confidence level of each tool for each type of defect requires: a) the confidence level assessment object; b) a benchmark dataset for testing; and c) a confidence level evaluation index system.

[0042] In this embodiment, four open-source static analysis tools were selected for fusion: cppcheck, flawfinder, clang, and codeql. The benchmark dataset used for testing was Juliet-test-suite. Although this dataset contains manually constructed code and cannot compare to real vulnerable code, it was the only option. However, when multiple static analysis tools are used in parallel to perform vulnerability detection on the program under test, the content, format, and specifications of the reports output by each tool are inconsistent, and there is overlap. Therefore, the detection reports need to be normalized, key information extracted, and similar items merged.

[0043] This embodiment also provides a confidence assessment system for state analysis tools. The assessment granularity is line-level code. The criteria for judgment are that the number of lines reported by the tool is correct and the defect type reported is consistent with the defect type of the test sample. Definition:

[0044] Positive TP report: The test case line x contains defect i, and the tool reports that line x contains defect i;

[0045] False positive (FP): There is no defect in line x of the test case, but the tool reports that there is defect i in line x;

[0046] FN Miss: The test case has defect i in line x, but the tool did not report it;

[0047] True negative TN: There is no defect in line x of the test case, and the tool does not report it;

[0048] Precision rate:

[0049] Recall rate:

[0050] F1 value:

[0051] Four tools were used to scan the benchmark dataset, and a confusion matrix was calculated according to the evaluation system. The F1 score was used as the confidence level to obtain the reliability of each tool's detection report for each type of code defect. The confidence level will play a role as a weighting factor during the tool fusion stage. The benchmark dataset has already labeled the location and type of defects. By comparing the benchmark results with the analysis tool results, the positive report rate, false positive rate, and accuracy rate of each tool for each type of defect can be determined. In this embodiment, the F1 score is selected as the confidence level of the tool for a certain type of defect; other indicators can also be used as the confidence level in this case.

[0052] Before performing vulnerability detection, it is necessary to perform function-level slicing of the code of the project under test and extract the function code attribute graph. Combined with the code attribute graph specification dictionary, the code attribute graph is represented to obtain the code attribute graph vector.

[0053] Specifically, a code attribute graph is a data structure composed of an abstract syntax tree, a control flow graph, and a program dependency graph, containing complete source code syntax information. By slicing a software project at the function level to obtain function-level code segments, these segments are input into the open-source code attribute graph generation tool Joern, which converts the source code into a code attribute graph. A code attribute graph is a directed, edge-labeled attributed multigraph. The code attribute graph specification clearly defines the graph pattern, the types and attributes of nodes and edges, and imposes constraints on source and target nodes. Using the code attribute graph specification, a code attribute graph dictionary is defined:

[0054]

[0055] The code attribute graph dictionary corpus elements consist of code attribute graph specifications and code attribute graph operators. There are 17 categories of code attribute graph specification tags, totaling 101 types, and 39 types of code attribute graph operators.

[0056] Given a function snippet, generate a code property graph g. z = (v, e), where v represents the set of vertices and e represents the set of edges. Parsing code attribute graph g z Traverse the nodes and edges of the graph, and put the encountered nodes and edges into a list. Assume the z function code property graph g... z Parsed as a node-edge sequence s z =[PARM,bIND, <operator>[,...], compare the node edge sequence with the dictionary corpus set, and calculate the relationship between each dictionary element and the sequence s. i The frequency of occurrence of an element in the dictionary corpus is recorded as follows: if the element appears in the sequence, it is marked as 1; otherwise, it is marked as 0. The final code attribute graph g is then used. z It can be represented as a 140-dimensional sparse vector. The algorithm for representing code attribute graph vectors is as follows:

[0057]

[0058] After obtaining the code attribute graph vector and the confidence scores of each static analysis, the detection reports can be fused. The fused representation of the analysis tool's detection report needs to consider both the tool's confidence score for the defect and the syntactic and semantic information of the source code function segment to obtain the feature vector of the report, such as... Figure 2 As shown:

[0059]

[0060] in, Let z be the code attribute graph vector, m*w ta m*w tb m*w tc m*w td This indicates whether static analysis tools a, b, c, and d have this test report. The value of m is 0 or 1, where 0 indicates no test report and 1 indicates a test report exists. ta w tb w tc w td These represent the confidence levels of defects y detected by static analysis tools a, b, c, and d, respectively. During the specific normalization process, based on the previously obtained tool confidence levels, analysis results with low confidence levels are discarded. For example, the confidence level of the CodeQL tool for defects like CWE121 is 0.1, therefore the result of the CodeQL tool identifying defect cwe121 will be discarded.

[0061] For example, if tools a and c report a defect y in line x of the z function of open-source software, then the representation vector of this report is: In this way, the feature vector contains both information from the context function and takes into account the confidence level of the tool.

[0062] Based on the analysis tool report fusion representation method proposed above, the same processing is applied to the benchmark dataset for training the classification model. Specifically, the benchmark dataset is divided into training and test sets according to the proportion. The training set is scanned using a static analysis tool to obtain detection reports. The report fusion representation method is used to convert the reports into feature vectors, and the fusion feature set is labeled with the benchmark data. The model is then trained under supervision to obtain a pre-trained model.

[0063] Finally, the feature vector obtained by fusing the detection reports of each static analysis tool corresponding to the project under test is input into the pre-trained model. The model determines whether the analysis result is a positive report or a false report. If the model predicts that the analysis result is a positive report, it is considered that there is a defect CWEXXX in the source code, and the vulnerability detection is completed.

[0064] It should be noted that, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances. The accompanying drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0065] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.< / operator>

Claims

1. A vulnerability discovery method based on the fusion of results from multiple static analysis tools, characterized in that, include: Step 1: Scan the benchmark dataset using multiple static analysis tools and calculate the confidence level of each static analysis tool based on the results; Step 2: Slice the project under test into function-level slices and extract the function code attribute graphs. Combine the code attribute graph specification dictionary to represent the code attribute graphs and obtain code attribute graph vectors. Step 3: Then, use multiple static analysis tools to test the project under test, and normalize and merge the test reports from each static analysis tool. Step 4: Combine the code attribute graph vector to represent the normalized and fused detection report as a feature vector; Step 5: Input the feature vector into the pre-trained classification model. If the classification model determines that the detection report is a positive report, it means that there is a vulnerability at the corresponding location. In step 2, the code attribute graph standard dictionary consists of 140 corpus elements. The vector representation process requires parsing and traversing the code attribute graph, putting the nodes and edges in the code attribute graph into the list sequence in turn, and finally comparing the list with the dictionary to calculate the frequency of dictionary elements appearing in the list sequence. In step 2, the code attribute graph vector is represented as follows: in, A dictionary for code property graphs. ,in Represents the set of vertices. Let be the set of edges; In step 3, when performing normalization fusion processing, the analysis results with low confidence are discarded based on the confidence level of the static analysis tool. In step 4, the feature vector is represented as follows: in, The code attribute graph vector for the z function. , , , This indicates whether static analysis tools a, b, c, and d have this test report. The value can be 0 or 1, where 0 indicates that there is no test report and 1 indicates that there is a test report. , , , These represent the confidence levels of the defects y detected by static analysis tools a, b, c, and d, respectively.

2. The vulnerability discovery method based on the fusion of results from multiple static analysis tools according to claim 1, characterized in that, In step 1, the benchmark dataset is Juliet-test-suite.

3. The vulnerability discovery method based on the fusion of results from multiple static analysis tools according to claim 1 or 2, characterized in that, In step 1, using a benchmark dataset and a confidence assessment system, the F1 score is calculated based on the line-of-code granularity of the static analysis tool's analysis results, and the F1 score is used as the confidence score of the corresponding static analysis tool.

4. The vulnerability mining method based on the fusion of results from multiple static analysis tools according to claim 3, characterized in that, The confidence level assessment system is specifically as follows: Positive TP report: The test case line x contains defect i, and the tool reports that line x contains defect i; False positive (FP): There is no defect in line x of the test case, but the tool reports that there is defect i in line x; FN Miss: The test case has defect i in line x, but the tool did not report it; True negative TN: There is no defect in line x of the test case, and the tool does not report it; Precision rate: ; Recall rate: ; F1 value: .

5. The vulnerability discovery method based on the fusion of results from multiple static analysis tools according to claim 1, characterized in that, The classification model training process is as follows: the benchmark dataset is divided into a training set and a test set according to a ratio; The training set is scanned using static analysis tools to obtain detection reports. The reports are then converted into feature vectors using a report fusion representation method. The fused feature set is labeled with benchmark data, and the model is trained under supervision.

Images