Vulnerability patch positioning method based on Bayesian optimization
By introducing Bayesian optimization into the RankNet model and dynamically adjusting the hyperparameters and feature weights, the problem of low accuracy and coverage in the vulnerability patch positioning method is solved, the sorting accuracy and recall rate are significantly improved, and more efficient vulnerability patch positioning is achieved.
Patent Information
- Application Number
- CN202510148253.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-11
- Publication Date
- 2025-06-10
AI Technical Summary
The existing vulnerability patch location methods have low accuracy and low coverage, making it difficult to effectively utilize vulnerability information to capture the semantic association between vulnerability description and code submission, and the sorting accuracy of the RankNet model is limited by fixed parameters and lack of dynamic adjustment mechanism.
Using a vulnerability patch location method based on Bayesian optimization, the hyperparameters of the RankNet model are dynamically adjusted, and the hyperparameter optimization is used to minimize the loss function, and the global average accuracy function MAP of the sorting result is adjusted using Bayesian optimization.
The sorting accuracy is significantly improved and the average accuracy of 10%-15%. After optimization, the proportion of CVE ranking first has increased from 69.49% to 74.13%. At the same time, automation optimization and accelerate convergence are accelerated, taking into account accuracy and recall, the MAP has increased from 0.78 to 0.84, and the recall has also increased from 85% to 92%.
Smart Images
Figure CN120124065A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of network security, and particularly relates to a method for locating vulnerability patches. Background Art
[0002] With the wide application of open-source software, vulnerability repair has become an important topic for maintaining system security, and security patches play a crucial role in the process of vulnerability repair. After a vulnerability is disclosed, developers need to locate relevant security patches in a timely manner, and the tens of thousands of code submissions make this work extremely difficult. Under the existing work, in order to avoid large-scale blind searches, researchers mainly locate by searching commit messages with CVE-ID, checking URLs of similar commits in CVE / NVD, and checking URLs with patch marks in NVD. These methods generally have the following deficiencies:
[0003] (1) Low accuracy and low coverage;
[0004] (2) Do not effectively utilize vulnerability information and it is difficult to capture the semantic association between vulnerability descriptions and code submissions;
[0005] (3) The model is based on matching, so the results obtained are few or none at all, and the results are directly equal to each other, which is not conducive to subsequent manual operations;
[0006] PatchScout, as an emerging vulnerability repair assistance tool, utilizes feature extraction to mine the application of the RankNet model in the association ranking of open-source project vulnerabilities - code submissions. However, the sorting accuracy of the RankNet algorithm of PatchScout is still limited by the following aspects:
[0007] (1) The parameters of the RankNet output probability are fixed and it is difficult to apply to different project and vulnerability characteristics;
[0008] (2) Lack of a dynamic adjustment mechanism for the feature weights of the sorting model;
[0009] Existing research shows that Bayesian optimization is a method suitable for black-box optimization, and can efficiently optimize parameters by modeling the objective function. Therefore, introducing Bayesian optimization into the RankNet sorting model can effectively improve accuracy and stability. Summary of the Invention
[0010] Aiming at the problem of insufficient accuracy in the process of sorting vulnerabilities - code submissions by the current RankNet model, the present invention proposes a method for locating vulnerability patches based on Bayesian optimization.
[0011] The vulnerability patch location method based on Bayesian optimization provided by the present invention, on the basis of PatchScout vulnerability information extraction and feature generation, and RankNet ranking to generate initial probabilities, dynamically adjusts the hyperparameters (such as the size of the hidden layer, the maximum number of iterations, etc.) of the RankNet model through Bayesian optimization; uses the Gaussian process model for hyperparameter optimization to minimize the loss function; after repeated iterations and parameter adjustments, finally obtains a sorting model with excellent performance; includes constructing the objective function of Bayesian optimization so that it can be fully applied to the vulnerability-code submission sorting based on the RankNet sorting model [1]; according to the output of the RankNet sorting model, uses Bayesian optimization to dynamically adjust the Mean Average Precision (MAP) of the sorting result; the specific steps are as follows:
[0012] Step 1: Data input;
[0013] Through information extraction and feature generation, a total of 22 relevant features in four categories, namely vulnerability identifier, vulnerability location, vulnerability type, and vulnerability description text, are used as input; among them:
[0014] Vulnerability identifier: Select CVE-ID and software-specific Bug-ID, a total of 2 features; CVE-ID is the public vulnerability reference, and the software-specific Bug-ID is the ID specifically assigned to the software before the CVE-ID is granted, which is used to track the life cycle of the vulnerability;
[0015] Vulnerability location: Use the file or function containing the file to describe the location of the vulnerability; count the number of files and irrelevant files mentioned in the submission, and at the same time calculate the ratio of the files shared by the vulnerability and the submission; for the function feature, count the number of functions and irrelevant functions mentioned in the submission, and at the same time calculate the ratio of the functions shared by the vulnerability and the submission; a total of 6 features;
[0016] Vulnerability type: Used to indicate the type and impact of the vulnerability; includes: the correlation between the vulnerability type of the vulnerability and the code submission, and the probability that the code submission is a security patch; a total of 2 features;
[0017] Vulnerability description text: Calculate the number, ratio, maximum shared frequency, total shared frequency, average shared frequency, and shared frequency weight of the shared words between the vulnerability description text and the submission message, and between the vulnerability description and the code difference analysis, with 6 features each, for a total of 12 features.
[0018] Step 2: RankNet sorting model sorting;
[0019] Based on the RankNet model, calculate the sorting score for the vulnerability-submission and generate a preliminary result. The specific process is as follows:
[0020] Step 2.1: For each group of submissions <Xi , X j >] to determine X i The probability of ranking ahead of X j is:
[0021]
[0022] X i , X j is the submission text to be sorted.
[0023] Step 2.2: According to the given object pairs, determine X i The learning probability of ranking ahead of X j is:
[0024]
[0025] S i , S j respectively represent the scores corresponding to the submission X i , X j .
[0026] Step 2.3: The cross-entropy loss function C is as follows:
[0027]
[0028] Step 2.4: Use the loss function to train the neural network (RankNet model) to minimize the loss on the training set;
[0029] Step 3: Bayesian optimization;
[0030] Use Bayesian optimization to dynamically adjust the RankNet algorithm and optimize the MAP value as follows:
[0031] Step 3.1: Optimization objective: Take the global average precision function MAP as the objective function and dynamically adjust the output probability distribution:
[0032] Define MAP as the objective function and optimize the RankNet output probability distribution:
[0033]
[0034] where Q represents the set of vulnerabilities, P(k) represents the precision of the top k rankings, rel(k) represents the relevance of the k-th submission (1 for relevant, 0 for irrelevant), and n represents the precision of calculating the top n rankings for each query;
[0035] Step 3.2: Optimization parameters: Automatically tune the feature weights, parameters, etc. of RankNet;
[0036] Step 3.3: Optimization method: Use Gaussian process to model the probability distribution of the RankNet output, and select the optimal parameters using the Expected Improvement (EI) strategy;
[0037] The specific process of the Bayesian optimization method is as follows:
[0038] (1) Initialize several feature sets, including feature weights, probability distribution parameters, and thresholds;
[0039] (2) Evaluate the RankNet sorting results based on the initial parameter set, calculate the MAP value, and record the corresponding relationship;
[0040] (3) Use Gaussian process to model the MAP value distribution and construct the MAP probability density function;
[0041] (4) Select the optimal parameter combination through the Expected Improvement (EI) strategy and iteratively update the MAP distribution;
[0042] The Expected Improvement (EI) strategy is expressed as:
[0043] EI(x) = E[max(f(x) - f * ), 0], (5)
[0044] where, f * is the current best MAP value; f(x) represents the MAP value of the RankNet model under the given parameter x.
[0045] (5) Stop the optimization when the MAP value converges or reaches the threshold.
[0046] Step 4: Result output
[0047] Through iterative optimization, output the sorted results after Bayesian optimization. The previous results are the security patches most likely to correspond to the vulnerabilities.
[0048] The technical advantages of the present invention are mainly as follows:
[0049] (1) Significantly improve the sorting accuracy.
[0050] After introducing Bayesian optimization, by dynamically adjusting the probability distribution of the RankNet output and the feature weights, the average sorting accuracy is increased by 10% - 15%. The proportion of ranking CVE in the first place after optimization is increased from 69.49% to 74.13%.
[0051] (2) Automatic optimization and accelerated convergence.
[0052] The RankNet model is applied to the probability distribution of vulnerability-code submission sorting results through Gaussian process modeling, and the expected improvement strategy is selected, resulting in an increased degree of automation, a significant reduction in the number of iterations, and a 10% acceleration in the average convergence time.
[0053] (3) Significantly enhance the global sorting performance.
[0054] Taking into account both precision and recall, the MAP increases from 0.78 to 0.84, and the recall rate also increases from 85% to 92%.
[0055] The method of the present invention significantly improves the precision and recall rate of the model, effectively narrows the search scope, significantly improves the convergence rate, demonstrates good performance in the problem of unbalanced datasets for security patch identification, and provides reliable technical support for open-source vulnerability patch location. Brief Description of the Drawings
[0056] Figure 1 It is a schematic diagram of the implementation process of the method of the present invention. Detailed Embodiment
[0057] The present invention selects five software that are currently widely used and have common vulnerabilities for simulation testing, including Linux kernel, Wireshark, QEMU, FFmpeg, and Jenkins.
[0058] Step 1: Vulnerability information extraction and feature generation:
[0059] Obtain CVE vulnerability information related to the corresponding software from the official website, 45 for each software, a total of 225 CVE vulnerability information; then through information extraction and feature generation, four categories of a total of 22 relevant features, namely vulnerability identifier, vulnerability location, vulnerability type, and vulnerability description text, are used as inputs, as shown in Table 1.
[0060] Table 1, Vulnerability feature extraction and generation format
[0061]
[0062]
[0063] Step 2: Create a test set
[0064] Find the corresponding patch submissions for the vulnerabilities in the test set and use them as the correct patches.
[0065] At the same time, add 450 non-security patch submissions and 325 security patch submissions for non-related vulnerabilities to the submission set to form interference items in the test set.
[0066] Correct security patch submission Non-security patch submission Irrelevant security patch submission Total number 225 450 325 。
[0067] Step 3: Apply the RankNet model for ranking
[0068] Use the RankNet model to perform a correlation ranking of the vulnerabilities and code submissions in the test set, obtain the patch correlation ranking results corresponding to each test vulnerability, and obtain the preliminary ranking results.
[0069] Check the relevant data output by the model, and count the frequency of correct security patches in the top 5 rankings as the basis for evaluating the ranking effect of the model before Bayesian optimization.
[0070] Base RankNet model sorting Number of successful first-place matches 156 Number of successful top-5 matches 191 .
[0071] Step 4: Apply the Bayesian-optimized RankNet model for ranking
[0072] Randomly select 10 vulnerabilities and corresponding patches that do not belong to the test set from five official software vulnerabilities, and randomly select 100 non-security patch submissions and 50 security submissions of non-related vulnerabilities as the training set.
[0073] Initialize the feature weights, probability distribution parameters, and thresholds as the initial parameter set for Bayesian optimization.
[0074] The learning rate is set to 0.01;
[0075] The maximum depth of the tree is set to 5 to prevent overfitting;
[0076] The regularization strength is set to 0.05;
[0077] The number of iterations is set to 100;
[0078] The number of cross-validation folds is set to 5.
[0079] Use the initial parameter set to evaluate the RankNet ranking results on the training set, calculate the MAP value and record the corresponding results.
[0080] Use the Gaussian process to model the MAP value distribution and construct the MAP probability density function.
[0081] Use the expected improvement (EI) strategy to select the parameter combination most likely to improve the MAP value, and update the model parameters according to the optimization results.
[0082] Iteratively optimize, automatically adjust and select the optimal hyperparameters through skopt.BayesSearchCV, optimize the output probability distribution and feature weights of the RankNet model, and improve the ranking accuracy and MAP value.
[0083] Finally, the entire test set is used as input to test the sorting effect of the optimized final RankNet model as follows:
[0084] Optimized RankNet model sorting Number of successful first-place matches 167 Number of successful top-5 matches 207
[0085] According to the above experimental results, under the Bayesian optimization of the present invention, the result of the vulnerability-code submission sorting algorithm based on the RankNet model is enhanced as follows:
[0086] (1) Significantly improve the sorting accuracy
[0087] After introducing Bayesian optimization, by dynamically adjusting the probability distribution and feature weights output by RankNet, the average precision of sorting is increased by 10%-15%. After optimization, the proportion of ranking CVE in the first place is increased from 69.49% to 74.13%.
[0088] Before optimization After optimization Number of successful first-place matches 156 167 Frequency of successful first-place matches 69.49％ 74.13％
[0089] (2) Automatic optimization and accelerated convergence
[0090] By modeling the probability distribution of the RankNet model applied to the vulnerability-code submission sorting result through Gaussian process, and the selection of the expected improvement strategy, the degree of automation is improved, the number of iterations is significantly reduced, and the average convergence time is accelerated by 10%.
[0091] (3) Significantly enhance the global sorting performance
[0092] Taking into account both precision and recall, MAP is increased from 0.78 to 0.84, and the recall rate is also increased from 85% to 92%.
[0093] Before optimization After optimization Number of successful top-5 matches 191 207 Top-5 match recall rate 85％ 92％ 。
[0094] References
[0095] [1]X. Tan, Y. Zhang, C. Mi, J. Cao, K. Sun, Y. Lin, and M. Yang, "Locating the security patches for disclosed OSS vulnerabilities with vulnerability-commit correlation ranking," Proc. 2021 ACM SIGSAC Conf. Computer and Communications Security (CCS'21), Virtual Event, Republic of Korea, 2021, pp. 3282-3299, doi: 10.1145 / 3460120.3484593。
Claims
1. A vulnerability patch location method based on Bayesian optimization, characterized in that: This includes constructing a Bayesian optimization objective function to fully apply it to the vulnerability-code submission ranking based on the RankNet ranking model; using Bayesian optimization to dynamically adjust the global average precision function MAP of the ranking results according to the output of the RankNet ranking model; the specific steps are as follows: Step 1: Data input; Through information extraction and feature generation, a total of 22 relevant features including vulnerability identifier, vulnerability location, vulnerability type and vulnerability description text are used as input; among them: Vulnerability Identifier: Select CVE-ID and software-specific Bug-ID, a total of 2 features; CVE-ID is the public vulnerability reference, and software-specific Bug-ID is the ID assigned to the software before the CVE-ID is granted, which is used to track the life cycle of the vulnerability; Vulnerability location: Use the file or function containing the file to describe the location of the vulnerability; count the number of files mentioned in the submission, the number of unrelated files, and calculate the ratio of vulnerabilities to submitted shared files; for function features, count the number of functions mentioned in the submission that are unrelated to the function, and calculate the ratio of vulnerabilities to submitted shared functions; a total of 6 features; Vulnerability type: used to indicate the type and impact of the vulnerability; including: the vulnerability type correlation between the vulnerability and the code submission, and the probability of the code submission being a security patch; a total of 2 features; Vulnerability description text: Analyze the shared words between vulnerability description text and submission message, vulnerability description and code difference, and calculate their number, proportion, maximum shared frequency, total shared frequency, average shared frequency and shared frequency weight, with 6 features each, for a total of 12 features. Step 2: Ranking by RankNet ranking model; Based on the RankNet model, the vulnerability submission is ranked and scored to generate preliminary results. The specific process is as follows: (1) For each set of submissions <X i ,X j >, judge X i Ranked in X j The previous probability: X i , X j The submission text to be sorted; (2) Based on the given object pair, determine X i Ranked in X j Previous learning probability: S i , S j Respectively represent submission X i , X j The corresponding ratings; (3) The cross entropy loss function C is as follows: (4) Use the loss function to train the neural network (RankNet model) to minimize the loss on the training set; Step 3: Bayesian optimization; Bayesian optimization is used to dynamically adjust the RankNet algorithm and optimize the MAP value, as follows: (1) Optimization objective: Taking the global average precision function MAP as the objective function, dynamically adjust the output probability distribution: Define MAP as the objective function to optimize the probability distribution of RankNet output: Where Q represents the vulnerability set, P(k) represents the top k precision, rel(k) represents the relevance of the kth submission, 1 represents relevant and 0 represents irrelevant, and n represents that for each query, the precision of the top n rankings is calculated; (2) Optimize parameters: Automatically tune the feature weights and parameters of RankNet; (3) Optimization method: Gaussian process is used to model the probability distribution of RankNet output, and the expected improvement (EI) strategy is used to select the optimal parameters; Step 4: Result output; Through iterative optimization, the Bayesian optimized sorting results are output, and the top results are the security patches that are most likely to correspond to the vulnerabilities.
2. The vulnerability patch location method based on Bayesian optimization according to claim 1 is characterized in that: The Bayesian optimization method has the following specific process: (1) Initialize several feature sets, including feature weights, probability distribution parameters and thresholds; (2) Evaluate the RankNet ranking results based on the initial parameter set, calculate the MAP value, and record the corresponding relationship; (3) Use Gaussian process to model the MAP value distribution and construct the MAP probability density function; (4) Select the most optimal parameter combination through the expected improvement (EI) strategy and iteratively update the MAP distribution; The expected improvement (EI) strategy is expressed as: EI(x)=E[max(f(x)-f * ),0] (5) Among them, f * is the current best MAP value; f(x) represents the MAP value of the RankNet model under the given parameter x; (5) Stop optimization when the MAP value converges or reaches the threshold.