Software vulnerability severity assessment method based on course learning and ordered regression
By adopting a method based on course learning and orderly regression in software vulnerability severity assessment, integrating bimodal information and using pre-trained models, the lag and precision problems of existing evaluation methods are solved, and high-precision vulnerability severity prediction and more reliable vulnerability management are achieved.
Patent Information
- Application Number
- CN202411733097.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-21
AI Technical Summary
The existing software vulnerability severity assessment methods have lag and insufficient evaluation accuracy, especially when resources are limited, it is difficult to quickly and accurately evaluate vulnerability severity and prioritize high-risk vulnerabilities.
Using a method based on course learning and orderly regression, the dual-modal information of source code and vulnerability description is integrated, the pre-trained model is efficiently utilized through the tip fine-tuning technology, and the data is gradually introduced according to the sample difficulty from easy to difficult for training through course learning, and the orderly relationship between severity levels is captured in combination with the orderly regression method.
It realizes high-precision prediction of the severity of software vulnerabilities, provides a more reliable vulnerability management basis, and significantly improves the accuracy and stability of vulnerability severity assessment.
Smart Images
Figure CN119670093A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a software vulnerability severity assessment method based on curriculum learning and ordered regression. Background Art
[0002] With the rapid development of information technology, software vulnerabilities have become a major hidden danger threatening computer system security and social and economic order. The diversity and rapid growth of vulnerabilities have made vulnerability management and repair face huge challenges, especially when resources are limited. How to quickly and accurately assess the severity of vulnerabilities and prioritize high-risk vulnerabilities has become a key issue.
[0003] Existing software vulnerability severity assessment methods mainly rely on expert knowledge and the Common Vulnerability Scoring System (CVSS). CVSS assesses the severity of vulnerabilities based on a series of qualitative and quantitative factors and provides a priority basis for vulnerability repair. However, the generation of CVSS severity scores usually requires manual analysis, which is lagging and difficult to meet actual vulnerability management needs in a timely manner. In addition, traditional assessment methods are mostly classification models that treat the severity levels of vulnerabilities as unordered categories and ignore the ordered relationship between severity levels, which limits the performance of the model in terms of assessment accuracy.
[0004] In recent years, pre-trained language models (such as CodeT5) have demonstrated excellent capabilities in code analysis and natural language processing tasks, bringing new possibilities to the field of vulnerability assessment. Unlike traditional classification models based on manual features, pre-trained language models can automatically capture semantic information in source code and vulnerability descriptions. However, there are still two important difficulties in applying pre-trained language models to vulnerability severity assessment: first, directly fine-tuning pre-trained models requires large-scale annotated data, and the cost of data acquisition is high in actual scenarios; second, traditional fine-tuning methods are difficult to fully utilize the knowledge learned in the pre-training stage, which may cause the model to be unable to effectively adapt to small sample tasks. However, most of the current research is based on a single modality (such as source code or vulnerability description) for vulnerability assessment, and fails to fully utilize the complementarity of the two. In addition, existing methods fail to consider the difference in sample difficulty, and directly adding complex samples to training may increase the difficulty of model convergence. Therefore, it is necessary to study a more advanced software vulnerability severity assessment method to better solve the problem of software vulnerability assessment.
[0005] How to solve the above technical problems becomes the subject faced by the present invention. Summary of the invention
[0006] The purpose of the present invention is to provide a software vulnerability severity assessment method based on curriculum learning and ordinal regression, which can predict the severity of software vulnerabilities.
[0007] The idea of the present invention is as follows: The present invention proposes a software vulnerability severity assessment method based on curriculum learning and ordinal regression. This method integrates bimodal information of source code and vulnerability descriptions, efficiently utilizes pre-trained models using prompt tuning technology, and gradually introduces data for training from easy to difficult according to sample difficulty through curriculum learning. Combining the ordinal regression method to capture the sequential relationship between severity levels, thereby achieving high-precision prediction of vulnerability severity and providing a more reliable basis for vulnerability management.
[0008] The present invention is implemented by the following measures: A software vulnerability severity assessment method based on curriculum learning and ordinal regression, which includes the following steps:
[0009] (1) Obtain records containing CVE ID, vulnerability description, and severity level from the CVE database. For each record, parse the open-source code libraries related to the vulnerability in the reference link, collect the CVE ID, pre-patch code, vulnerability severity, and vulnerability description of the vulnerability from them, and preprocess the extracted source code, including removing comments, blank lines, and leading spaces, retaining the core code logic, ensuring that the code structure is clear and can fully reflect the vulnerability characteristics, and organizing all data into a unified format: <CVE ID, pre-patch code, vulnerability description, vulnerability severity level>, forming a high-quality dataset D;
[0010] (2) Divide the dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1, and use stratified sampling to ensure that the vulnerability severity distribution in different sets is consistent;
[0011] (3) Design a curriculum learning framework based on the training data, including a difficulty measurer and a training scheduler. Assign difficulty scores to the data through the difficulty measurer, sort them from easy to difficult according to the scores, and sequentially introduce the sorted samples into the model for training;
[0012] (4) Construct a mixed prompt template, fuse the vulnerability description and the source code into a bimodal input, input it into the pre-trained model CodeT5, and train the software vulnerability severity assessment model in combination with the ordinal regression framework;
[0013] (5) During the model training process, adopt an early stopping strategy, stop training when the performance metrics of the validation set do not improve for multiple consecutive rounds, and finally return the optimal software vulnerability severity assessment model and the prompt template;
[0014] (6) For the software vulnerability to be evaluated, use the vulnerability description and the source code as inputs, combine them with the mixed prompt template, and input them into the trained software vulnerability severity assessment model to complete the automatic prediction of the vulnerability severity.
[0015] Further, in the step (3), it specifically includes the following steps:
[0016] (3-1) The training set is divided into four subsets, one of which is used as the validation set each time, and the remaining data is used as the training set for cross-validation to obtain four trained models;
[0017] (3-2) Use the trained model to predict the samples in the validation set, calculate the prediction error of the samples, and use the error value as the initial difficulty score of the samples;
[0018] (3-3) Summarize the difficulty scores of multiple subsets and generate a list of difficulty scores of all training samples;
[0019] (3-4) Sort the training samples from easy to difficult according to the difficulty scores, and generate a sorted sample queue for subsequent course learning scheduling;
[0020] (3-5) Based on the sorted sample queue, define a training scheduler to introduce samples in stages for model training: in the initial stage, only use samples with small errors between the predicted results and the actual results obtained in step (3-4) for training to help the model master the basic pattern; as the training process progresses, gradually introduce samples with large errors between the predicted results and the actual results to ensure that the model gradually learns the characteristics of complex samples on the basis of mastering the characteristics of simple samples; each time a new difficult sample is introduced, keep the sample data learned in the previous stage to improve the generalization ability of the model.
[0021] Furthermore, the step (4) specifically includes the following steps:
[0022] (4-1) Construct a hybrid prompt template, distinguish the vulnerability description and source code information through hard prompts, such as adding "The vulnerability description:" and "The code snippet:" prompts respectively; for severity assessment, use the [SOFT] tag to create soft prompts, such as initializing it to "Classify the severity:", and use the [MASK] tag to predict or fill in missing information. The specific format is:
[0023] The code snippet: [X] The vulnerability description: [Y] [SOFT] [MASK], where [X] indicates the source code and [Y] indicates the vulnerability description;
[0024] (4-2) Design a mapping method to map each predicted label of the model to a class in the target class set. For example, when the predicted label is "slight" or "low", it is mapped to the LOW class; when the predicted label is "medium" or "moderate", it is mapped to the MEDIUM class; when the predicted label is "severe" or "high", it is mapped to the HIGH class; when the predicted label is "significant" or "critical", it is mapped to the CRITICAL class;
[0025] (4-3) Training is based on the binary cross entropy loss function, which is defined as: Where N is the number of samples, C is the number of categories, is the binary label of sample i corresponding to category j. If the sample belongs to or exceeds category j, it is 1, otherwise it is 0; σ(z ij ) is the probability of category j predicted by the model.
[0026] Furthermore, the step (5) specifically includes the following steps:
[0027] (5-1) Calculate the changing trend of the MSE indicator on the validation set;
[0028] (5-2) An early stopping strategy is adopted. When the performance does not improve after 10 consecutive rounds of training, the training scheduler is used to introduce difficult data for training. The above steps (5-1) and (5-2) are repeated until the entire data set is trained and the optimal model and template are returned.
[0029] (6) For the software vulnerabilities that need to be evaluated, the vulnerability description and source code are used as input, combined with the mixed prompt template, and input into the trained model to complete the automatic prediction of the vulnerability severity.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes an ordered regression software vulnerability severity assessment method based on course learning improvement, fully utilizes the advantages of the pre-trained language model CodeT5, and realizes efficient model adaptation through the prompt fine-tuning method. This method integrates the bimodal information of vulnerability description and source code, and designs a hybrid prompt template, which not only improves the integrity of information expression, but also is closer to the task characteristics of the pre-trained model. In addition, the present invention introduces a course learning strategy, gradually introduces training data from easy to difficult according to the difficulty of the samples, optimizes the learning path of the model, and avoids complex samples from interfering with the model training process. By combining the ordered regression framework, the sequential relationship of the severity levels is captured, which effectively improves the accuracy and stability of the vulnerability severity assessment, and provides a more efficient and reliable solution for automated vulnerability management. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention.
[0032] Figure 1 It is a system framework diagram of the software vulnerability severity assessment method based on curriculum learning and ordinal regression provided by the present invention. Detailed implementation manners
[0033] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0034] Embodiment 1
[0035] Refer to Figure 1 As shown, this embodiment provides a software vulnerability severity assessment method based on curriculum learning and ordinal regression, which specifically includes the following content:
[0036] (1) Obtain records containing CVE ID, vulnerability description and severity level from the CVE database. For each record, parse the open-source code libraries related to the vulnerability in the reference link, collect the CVE ID, pre-fix code, vulnerability severity and vulnerability description of the vulnerability from them, and preprocess the extracted source code, including removing comments, blank lines and leading spaces, retaining the core code logic, ensuring that the code structure is clear and can fully reflect the vulnerability characteristics, and organizing all data into a unified format: <CVE ID, pre-fix code, vulnerability description, vulnerability severity level>, to form a high-quality data set D;
[0037] (2) Divide the data set into a training set, a validation set and a test set according to the ratio of 8:1:1, and use stratified sampling to ensure that the vulnerability severity distributions in different sets are consistent;
[0038] (3) Design a curriculum learning framework based on the training data, including a difficulty measurer and a training scheduler. Assign difficulty scores to the data through the difficulty measurer, sort the samples from easy to difficult according to the scores, and introduce the sorted samples into the model training in turn, which specifically includes the following steps:
[0039] (3-1) Divide the training set into four subsets, each time use one of the subsets as the validation set, and use the remaining data as the training set for cross-validation to obtain four trained models;
[0040] (3-2) Use the trained model to predict the samples in the validation set, calculate the prediction error of the samples, and use the error value as the initial difficulty score of the samples;
[0041] (3-3) Summarize the difficulty scores of multiple subsets and generate a list of difficulty scores of all training samples;
[0042] (3-4) Sort the training samples from easy to difficult according to the difficulty scores, and generate a sorted sample queue for subsequent course learning scheduling;
[0043] (3-5) Based on the sorted sample queue, define a training scheduler to introduce samples in stages for model training: in the initial stage, only use samples with small errors between the predicted results and the actual results obtained in step (3-4) for training to help the model master the basic pattern; as the training process progresses, gradually introduce samples with large errors between the predicted results and the actual results to ensure that the model gradually learns the characteristics of complex samples on the basis of mastering the characteristics of simple samples; each time a new difficult sample is introduced, keep the sample data learned in the previous stage to improve the generalization ability of the model.
[0045] (4) Construct a hybrid prompt template, fuse the vulnerability description and source code into a bimodal input, input it into the pre-trained model CodeT5, and combine it with the ordered regression framework to train the software vulnerability severity assessment model. The specific steps include:
[0046] (4-1) Construct a hybrid prompt template, distinguish the vulnerability description and source code information through hard prompts, such as adding "The vulnerability description:" and "The code snippet:" prompts respectively; for severity assessment, use the [SOFT] tag to create soft prompts, such as initializing it to "Classify the severity:", and use the [MASK] tag to predict or fill in missing information. The specific format is:
[0047] The code snippet: [X] The vulnerability description: [Y] [SOFT] [MASK], where [X] indicates the source code and [Y] indicates the vulnerability description;
[0048] (4-2) Design a mapping method to map each predicted label of the model to a class in the target class set. For example, when the predicted label is "slight" or "low", it is mapped to the LOW class; when the predicted label is "medium" or "moderate", it is mapped to the MEDIUM class; when the predicted label is "severe" or "high", it is mapped to the HIGH class; when the predicted label is "significant" or "critical", it is mapped to the CRITICAL class;
[0049] (4-3) Training is based on the binary cross entropy loss function, which is defined as: Where N is the number of samples, C is the number of categories, is the binary label of sample i corresponding to category j. If the sample belongs to or exceeds category j, it is 1, otherwise it is 0; σ(z ij ) is the probability of category j predicted by the model.
[0050] (5) During the model training process, an early stopping strategy is adopted. When the performance indicators of the validation set do not improve after multiple consecutive rounds, the training is stopped. Finally, the optimal software vulnerability severity assessment model and prompt template are returned. Specifically, the following steps are included:
[0051] (5-1) Calculate the changing trend of the MSE indicator on the validation set;
[0052] (5-2) An early stopping strategy is adopted. When the performance does not improve after 10 consecutive rounds of training, the training scheduler is used to introduce difficult data for training. The above steps (5-1) and (5-2) are repeated until the entire data set is trained and the optimal model and template are returned.
[0053] (6) For the software vulnerabilities that need to be evaluated, the vulnerability description and source code are used as input, combined with the mixed prompt template, and input into the trained model to complete the automatic prediction of the vulnerability severity.
[0054] (7) The method of this embodiment is compared with the existing software vulnerability assessment method on the same data set, and two performance indicators (i.e., MSE and MZE) from ordered regression are used to automatically evaluate the quality of the model:
[0055] Table 1 Comparison of results between the method of this embodiment and other methods
[0056]
[0057] After experimental verification, the improved ordered regression software vulnerability severity assessment method based on curriculum learning proposed in this embodiment is significantly better than the corresponding best benchmark results in both MSE and MZE key performance indicators compared to the existing benchmark methods, showing the outstanding performance of the method of this embodiment in vulnerability severity prediction. Specifically, the experimental results are as follows: in terms of MSE, the value of the method of this embodiment is 0.295, which is 45% less than the best performing Fun-RF (0.535); in terms of MZE, the value of the method of this embodiment is 0.261, which is 24% higher than the best performing CWM-XGB (0.343). These results show that the improved ordered regression method based on curriculum learning can significantly improve the accuracy of vulnerability severity assessment, especially when dealing with complex vulnerability information, and has stronger robustness and predictive ability than existing methods.
[0058] Example 2
[0059] On the basis of Example 1, the data set is re-divided, and the method of this embodiment is compared with the existing software vulnerability severity assessment method. Two performance indicators (i.e., MSE and MZE) from ordered regression are used to automatically evaluate the quality of the model:
[0060] Table 2 Comparison of results between the method of this embodiment and other methods
[0061]
[0062] After experimental verification, the improved ordered regression software vulnerability severity assessment method based on curriculum learning proposed in this embodiment is significantly better than the corresponding best benchmark results in both MSE and MZE key performance indicators compared to the existing benchmark methods, showing the outstanding performance of the method of this embodiment in vulnerability severity prediction. Specifically, the experimental results are as follows: in terms of MSE, the value of the method of this embodiment is 0.295, which is 43% less than the best performing Fun-RF (0.547); in terms of MZE, the value of the method of this embodiment is 0.2, which is 23% higher than the best performing CWM-XGB (0.355). These results show that the improved ordered regression method based on curriculum learning can significantly improve the accuracy of vulnerability severity assessment, especially when dealing with complex vulnerability information, and has stronger robustness and predictive ability than existing methods.
[0063] Example 3
[0064] Based on Example 1, the data set is re-divided, the method of this embodiment is compared with the existing software vulnerability severity assessment method, and two performance indicators (i.e., MSE and MZE) from ordered regression are used to automatically evaluate the quality of the model.
[0065] Table 3 Comparison of results between the method of this embodiment and other methods
[0066]
[0067] After experimental verification, the improved ordered regression software vulnerability severity assessment method based on curriculum learning proposed in this embodiment is significantly better than the corresponding best benchmark results in both MSE and MZE key performance indicators compared to the existing benchmark methods, showing the outstanding performance of the method of this embodiment in vulnerability severity prediction. Specifically, the experimental results are as follows: in terms of MSE, the value of the method of this embodiment is 0.305, which is 41% less than the best performing Fun-RF (0.520); in terms of MZE, the value of the method of this embodiment is 0.269, which is 23% higher than the best performing CWM-XGB (0.348). These results show that the improved ordered regression method based on curriculum learning can significantly improve the accuracy of vulnerability severity assessment, especially when dealing with complex vulnerability information, and has stronger robustness and predictive ability than existing methods.
[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A software vulnerability severity assessment method based on curriculum learning and ordinal regression, characterized in that: The steps include the following: (1) Obtain records containing CVE ID, vulnerability description, and severity level from the CVE database. For each record, parse the open-source code libraries related to the vulnerability in the reference link, collect the CVE ID, pre-fix code, vulnerability severity, and vulnerability description of the vulnerability from them, and preprocess the extracted source code, including removing comments, blank lines, and leading spaces, retaining the core code logic, ensuring that the code structure is clear and can fully reflect the vulnerability characteristics, and organizing all data into a unified format: <CVE ID, pre-fix code, vulnerability description, vulnerability severity level>, to form a high-quality dataset D; (2) Divide the dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1, and use stratified sampling to ensure that the vulnerability severity distribution in different sets is consistent; (3) Design a curriculum learning framework based on the training data, including a difficulty measurer and a training scheduler. Assign difficulty scores to the data through the difficulty measurer, sort them from easy to difficult according to the scores, and sequentially introduce the sorted samples into model training; (4) Construct a hybrid prompt template, fuse the vulnerability description and the source code into a bimodal input, input it into the pre-trained model CodeT5, and train a software vulnerability severity assessment model in combination with an ordered regression framework; (5) During the model training process, adopt an early stopping strategy, stop training when the performance metrics of the validation set do not improve for multiple consecutive rounds, and finally return the optimal software vulnerability severity assessment model and the prompt template; (6) For the software vulnerability to be evaluated, use the vulnerability description and the source code as inputs, combine them with the hybrid prompt template, and input them into the trained software vulnerability severity assessment model to complete the automatic prediction of the vulnerability severity.
2. A method for evaluating the severity of software vulnerabilities based on curriculum learning and ordinal regression according to claim 1, characterized in that: In step (3) above, the following steps are included: (3-1) Divide the training set into four subsets, each time use one of the subsets as the validation set, and use the remaining data as the training set for cross-validation to obtain four trained models; (3-2) Use the trained models to predict the samples in the validation set, calculate the prediction error of the samples, and use the error value as the initial difficulty score of the samples; (3-3) Aggregate the difficulty scores of multiple subsets to generate a difficulty score list for all training samples; (3-4) Sort the training samples from easy to difficult according to the difficulty scores to generate a sorted sample queue for subsequent curriculum learning scheduling; (3-5) Based on the sorted sample queue, define a training scheduler to introduce samples in stages for model training: in the initial stage, only use the samples with small errors between the prediction results and the true results obtained in step (3-4) for training to help the model master the basic mode; as the training process progresses, gradually introduce the samples with large errors between the prediction results and the true results to ensure that the model gradually learns the characteristics of complex samples on the basis of mastering the characteristics of simple samples; When introducing new difficult samples each time, retain the sample data learned in the previous stage to improve the generalization ability of the model.
3. The method for evaluating the severity of software vulnerabilities based on curriculum learning and ordinal regression according to claim 1, characterized in that: In step (4) above, the following steps are included: (4-1) Construct a hybrid prompt template, distinguish the vulnerability description and source code information through hard prompts, and add "The vulnerability description:" and "The code snippet:" prompts respectively; for severity assessment, use the [SOFT] tag to create a soft prompt, initialized to "Classify the severity:", and use the [MASK] tag to predict or fill in missing information. Its format is: The code snippet: [X] The vulnerability description: [Y] [SOFT] [MASK], where [X] indicates the source code and [Y] indicates the vulnerability description; (4-2) Design a mapping method to map each predicted label of the model to a class in the target class set. When the predicted label is "slight" or "low", it is mapped to the LOW class; when the predicted label is "medium" or "moderate", it is mapped to the MEDIUM class; when the predicted label is "severe" or "high", it is mapped to the HIGH class; when the predicted label is "significant" or "critical", it is mapped to the CRITICAL class; (4-3) Training is based on the binary cross entropy loss function, which is defined as: Where N is the number of samples, C is the number of categories, is the binary label of sample i corresponding to category j. If the sample belongs to or exceeds category j, it is 1, otherwise it is 0; σ(z ij ) is the probability of category j predicted by the model.
4. The method for evaluating the severity of software vulnerabilities based on curriculum learning and ordinal regression according to claim 1, characterized in that: The step (5) comprises the following steps: (5-1) Calculate the changing trend of the MSE indicator on the validation set; (5-2) An early stopping strategy is adopted. When the performance does not improve after 10 consecutive rounds of training, the training scheduler is used to introduce difficult data for training. The above steps (5-1) and (5-2) are repeated until the entire data set is trained and the optimal model and template are returned.