Vulnerability severity score prediction method based on large language model and retrieval enhancement

Abstract

The invention belongs to the technical field of network security, and particularly relates to a vulnerability severity score prediction method based on a large language model and retrieval enhancement. According to the method, an automatic scoring framework integrating data preprocessing, hierarchical CWE classification and an RAG technology is constructed; firstly, redundant noise of vulnerability description is eliminated through a version number cleaning algorithm driven by a regular rule, and then domain knowledge modeling is enhanced in combination with a hierarchical CWE classifier based on MITRE View-1003; and an RAG technology is adopted to fuse semantic features of historical similar vulnerabilities and a large language model generation capability, so that high-precision and interpretable CVSS score prediction is realized. Experiments show that the method can be widely applied to severity score prediction of various types of vulnerabilities, has good prediction robustness and effectiveness, significantly improves vulnerability management efficiency and decision transparency, and provides an effective tool for evaluation and management of software supply chain security.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a vulnerability severity score prediction method. Background Art

[0002] With the increasing complexity of software systems, vulnerability management has become increasingly important. Currently, the Common Vulnerability Scoring System (CVSS) is widely used for vulnerability risk assessment. However, due to the lag, subjectivity, and inefficiency of manual scoring, the demand for automated vulnerability scoring is increasing. Existing automated tools face certain technical bottlenecks in terms of semantic understanding of vulnerability descriptions, utilization of domain knowledge, and interpretability of scoring results. Most existing methods rely on traditional machine learning or deep learning models, but they are easily interfered with by redundant information when processing complex vulnerability descriptions and lack sufficient consideration of historical vulnerability cases. As a result, they have significant deficiencies in the accuracy and interpretability of vulnerability severity predictions.

[0003] CVSS severity score prediction method Summary of the Invention

[0004] The purpose of this invention is to provide a vulnerability severity scoring prediction method based on a large language model and retrieval enhancement with high vulnerability scoring accuracy and strong interpretability, so as to significantly improve the efficiency of vulnerability management and decision-making transparency, and effectively solve the bottlenecks of existing automated tools in semantic understanding, domain knowledge and interpretability, thereby providing an effective tool for the assessment and governance of software supply chain security.

[0005] The vulnerability severity score prediction method based on large language model and retrieval enhancement (abbreviated as CVSS severity score prediction method) provided by this invention integrates data preprocessing, CWE level classification and retrieval enhancement generation (RAG) technology; it includes four stages (i.e., four steps): data preprocessing, CWE level classification, retrieval enhancement generation, and score prediction and optimization: Figure 1 As shown, where:

[0006] (1) In the data preprocessing stage, redundant information in the vulnerability description is removed to improve the model's ability to understand the vulnerability description and optimize data quality. Specifically:

[0007] Redundant text information in vulnerability descriptions, such as version numbers and manufacturer names, is cleaned and standardized and denoised using algorithms such as regular expressions to ensure that the technical features in the vulnerability descriptions are more prominent and reduce the impact of irrelevant information. The text is also semantically standardized using techniques such as morphological restoration and stemming to ensure semantic consistency.

[0008] (2) In the CWE hierarchical classification stage, based on the View-1003 classification view in the CWE classification system proposed by the MITRE organization (see CWE-1003: Weaknesses for Operational Cybersecurity View on the MITRE official website), a cross encoder is used to jointly optimize the top-level coarse-grained classification and the bottom-level fine-grained classification to accurately predict the CWE category to which the vulnerability belongs, providing more accurate weakness category annotation for the prediction of CVSS scores.

[0009] (3) In the retrieval-augmented generation (RAG) stage, the retrieval model is used to search for vulnerability records similar to the current vulnerability description in the historical vulnerability database through dense vector retrieval technology, and the semantic similarity between the vulnerability description and the historical records is calculated; the large language model is used to generate the CVSS score, and the generated score is interpreted in combination with the historical vulnerability information to ensure the traceability of the scoring process and the transparency of the decision.

[0010] (4) In the scoring prediction and optimization stage, the accuracy of the model's prediction results is evaluated, and the model parameters are adjusted through experimental data to make the scoring results more accurate and improve the model's generalization ability on different data sets.

[0011] Further:

[0012] In step (1), the vulnerability description is preprocessed by cleaning the vulnerability description through text noise reduction and standardization, version number detection and replacement, separation of product name and version number, removal of irrelevant version numbers, etc., and the version number rule set is expanded into 14 major categories and 67 subcategories according to different basic types. These rule sets cover more version number types and further refine various rules.

[0013] The specific process is:

[0014] (1.1) Text noise reduction and standardization. First, based on the stop word list in the NLP library (such as "and", "the", "for", etc.), high-frequency words that contribute little to semantic understanding are removed to reduce the interference of irrelevant features. All texts are uniformly converted to lowercase to eliminate vocabulary redundancy caused by inconsistent capitalization. At the same time, special characters, HTML tags, redundant spaces and irrelevant punctuation marks are removed to reduce data noise. Since vulnerability descriptions often contain different tenses or word form variants, we use word form normalization methods to convert words into their basic word forms. For example, "running" is converted to "run" to avoid different forms of the same word being mistakenly identified as different concepts. In addition, stem extraction technology can be combined to trim words to their basic roots, such as simplifying "programming" to "program".

[0015] (1.2) Version number detection and replacement. Use regular expressions to match common version number formats and replace them with standardized placeholders, such as replacing "1.2.3" with "[VERSION]." This approach effectively prevents version number information from interfering with subsequent semantic modeling.

[0016] (1.3) Product name and version number separation. In some vulnerability descriptions, the version number and product name are closely combined, such as "Microsoft Windows 10.0.19041." We use a lexicon-based splitting method to separate the product name from the version number, retaining only the product name. This ensures that the vulnerability description focuses on the technology.

[0017] (1.4) Remove irrelevant version numbers. When multiple version numbers appear in a description, such as "affects versions 2.0, 2.1, 2.3, and 2.4", we normalize the expression, retain the main version information, and remove redundant enumeration content.

[0018] (1.5) Based on these basic types, the version number rule set is expanded into 14 major categories, including 67 subcategories. The following table lists the specific rule sets designed by this invention for the standardization of version number information. These rule sets cover more version number types and further refine each type of rule.

[0019]

[0020] In step (2), the CWE hierarchical classification is achieved by jointly optimizing the top-level and bottom-level classifiers. By training a cross-encoder, the top-level coarse-grained classification and the bottom-level fine-grained classification are jointly optimized to enhance the recognition ability of various features in the vulnerability description, thereby improving the accuracy of CWE prediction.

[0021] First, MITRE's CWE classification tree (organized according to View-1003, containing 37 first-level top-level nodes, 93 second-level bottom-level nodes, and a total of 130 classification nodes) is used to separate the top and bottom levels, and a corresponding training data set is constructed based on the hierarchical information. Then, a "hierarchical classifier" is used to determine whether the input CVE description should belong to the top or bottom level. Subsequently, within the determined hierarchical range, the "top-level classifier" and "bottom-level classifier" are used to perform specific CWE predictions (in order to make full use of the text semantic features, necessary text cleaning and data enhancement operations are performed in the data preparation stage to maximize the mining of key information in the CVE description); the specific process is as follows ( Figure 2 shown):

[0022] (2.1) Classification by hierarchical classifier. The input of the hierarchical classifier is the preprocessed CVE description text. After contextual semantic modeling using the Secure-BERT model (Secure-BERT is a BERT variant model trained on cybersecurity corpus, and the relevant content has been published in "Securebert: A domain-specific language model for cybersecurity"), the CLS representation of the text is extracted and input into a fully connected classification layer for binary classification. Ultimately, the model outputs a binary label, indicating whether the text belongs to the top layer (label "1") or the bottom layer (label "0"). In the data preprocessing stage, we adopted strategies such as text denoising, synonym replacement, and version number normalization to minimize redundancy and noise interference, thereby improving the classification performance of the model. Thanks to Secure-BERT's effective integration of security domain knowledge, the hierarchical classifier can achieve more robust classification results in the dimension of hierarchical decision-making. At the same time, the preprocessing strategy also provides important support for improving the performance of the model.

[0023] (2.2) Classification by top-level and bottom-level classifiers. After completing the hierarchical classification, if the CVE description is predicted to be top-level (TopLayer), it is sent to the top-level classifier; if it is predicted to be bottom-level (Bottom Layer), it is sent to the bottom-level classifier. Both use the Cross-Encoder model (the Cross-Encoder model is a deep dual-tower structure widely used in text pair semantic matching tasks, and the relevant content has been published in "Sentence-bert: Sentence embeddings using siamese bert-networks") to calculate the semantic similarity score of the "CVE description-CWE information" string pair to identify the most likely CWE category.

[0024] The Cross-Encoder model receives two text inputs (CVE description and CWE text), concatenates them, and feeds them into the BERT model. The Transformer layer captures the contextual associations and semantic dependencies between the two texts. The specific process is as follows:

[0025] For each CVE description, all CWE entries corresponding to the top or bottom classifier are combined to generate a string pair; the string pair is input into the Cross-Encoder model, and the semantic similarity score of each pair of texts is calculated (ranging from 0 to 1); the text is sorted in descending order according to the semantic similarity score, and the CWE entry with the highest score is selected as the final prediction result.

[0026] The Cross-Encoder's prediction results not only directly output the most likely CWE category but also provide a complete list of similarity scores. This output enhances the model's interpretability and facilitates further verification and analysis of the prediction results by security experts.

[0027] In step (3), the retrieval enhancement generation (RAG) is achieved through the following steps: similar vulnerability retrieval, using a two-level retrieval scheme of "Bi-Encoder (coarse retrieval) + Cross-Encoder (fine retrieval)". This mechanism can not only quickly screen candidate documents from massive data, but also perform deep semantic matching on candidate results to ensure retrieval accuracy; after completing the similar vulnerability retrieval, the model obtains K2 historical vulnerability cases that are most similar to the target vulnerability description. Then, through the deep learning-based retrieval model, the vulnerability description is converted into a dense vector, and the vulnerability case with the most similar semantics to the description is searched in the historical vulnerability database; the retrieval enhancement generation (RAG) is performed through the large language model (LLM) fine-tuned by Prompt to generate a CVSS score, and the generated result is further inferred and corrected in combination with the retrieved similar vulnerability information to improve the accuracy of the inference of various metric values ​​in the CVSS vector.

[0028] The technical features and beneficial effects of the present invention mainly include:

[0029] The present invention demonstrates significant advantages in CVSS score prediction by combining data cleaning, CWE hierarchical classification and RAG technology. Experimental results show that when this method is tested on a dataset of 19,794 NVD vulnerabilities, the accuracy of the CVSS basic score reaches 71.45%, an improvement of 20.91 percentage points over the baseline model. In addition, the hierarchical CWE prediction module can accurately predict 130 CWE categories, with an F1-score of 73%, demonstrating good performance. In terms of interpretability, the retrieval enhancement generation mechanism provides a Top-3 historical vulnerability tracing success rate of up to 83.2%, greatly improving the efficiency of vulnerability governance and the transparency of decision-making. The present invention not only improves the accuracy of vulnerability scoring, but also enhances the verifiability and transparency of vulnerability scoring results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 FIG. 4 is a flowchart of the CVSS severity score prediction method of the present invention.

[0031] Figure 2 FIG. 4 is a flowchart of the CWE-ID classification method of the present invention. DETAILED DESCRIPTION

[0032] This paper proposes a method for predicting CVSS vulnerability severity scores based on a large language model and Retrieval Enhanced Generation (RAG) technology. This method primarily involves data preprocessing, CWE classification, Retrieval Enhanced Generation (RAG), and score prediction and optimization. The following sections describe the implementation and technical details of each module.

[0033] 1. Data Preprocessing

[0034] Data preprocessing is a crucial step in this invention, as it directly affects the accuracy and effectiveness of subsequent models. This module cleans and normalizes vulnerability description data through the following steps:

[0035] (1) Noise data removal: Vulnerability descriptions often contain redundant information such as version numbers, manufacturer information, and operating system information that is irrelevant to the technical characteristics of the vulnerability. To remove this noise, the present invention designs a rule engine based on regular expressions. The engine uses regular rules such as:

[0036] (\d+\.\d+\.\d+) is used to match the version number format and replace it with the unified placeholder "[VERSION]", such as replacing v1.2.3 with [VERSION].

[0037] Clean up by using regular rules designed for specific vendor information (such as Windows, Cisco), making vulnerability descriptions more concise and focusing on technical features.

[0038] (2) Semantic standardization: Vulnerability descriptions contain a variety of terms and expressions. To unify expressions, we use morphological restoration technology. For example, "exploitability" and "exploitable" are standardized to "exploit". In addition, we remove stop words (such as "and" and "the") to reduce unnecessary noise. Stemming methods (such as converting "programming" to "program") are also applied to further improve text consistency.

[0039] (3) Version number normalization: Vulnerability descriptions often contain version numbers in various formats, such as v1.2.3 and 1.2.3-beta. This paper designs a specialized version number cleaning algorithm that uses regular expressions to identify version numbers in different formats and replaces them with a unified placeholder "[VERSION]". This allows the model to focus on the technical features of the vulnerability description and avoid the interference caused by the version number.

[0040] The code flow is shown in Appendix 1.

[0041] 2. CWE Tier Classification

[0042] CWE classification is one of the core modules of this invention. By accurately classifying vulnerability descriptions, we can better understand the technical characteristics of the vulnerability and provide more accurate basic data for CVSS scoring. The specific implementation steps are as follows:

[0043] (1) Using the CWE Hierarchy: This paper uses the MITRE View-1003 classification system, which contains 130 vulnerability categories organized in a hierarchical structure. By comparing the vulnerability description with each category in the hierarchy, the model can identify the corresponding category based on the context of the text. For example, for the "SQL Injection" vulnerability, the model will classify it as CWE-89 (SQL Injection Vulnerability).

[0044] (2) Adopting a cross-encoder model: In order to accurately predict the CWE category of the vulnerability, the present invention adopts a cross-encoder model. The cross-encoder is a bidirectional encoder architecture that can encode vulnerability descriptions and CWE categories at the same time. During the training process, we jointly encode each vulnerability description with multiple CWE categories and calculate the semantic similarity between them. In this way, the model can learn the fine-grained association between vulnerability descriptions and CWE categories and effectively improve the accuracy of classification. For example, for a CVSS description, "SQL injection vulnerability", the model will compare it with all related items in CWE and select the classification item with the highest similarity. Through this method, the model can more accurately identify specific CWE categories.

[0045] (3) Hierarchical Optimization: In the CWE hierarchical classification, the top-level classifier is used to perform coarse-grained classification of vulnerabilities, such as distinguishing between "injection vulnerabilities" and "cross-site scripting vulnerabilities." The bottom-level classifier is refined into specific vulnerability types, such as "SQL injection" or "XSS vulnerability." By jointly optimizing the top-level and bottom-level classifiers, the present invention can achieve multi-level precise classification, thereby improving the accuracy of CWE category prediction.

[0046] The code is shown in Appendix 2.

[0047] 3. Retrieval Enhanced Generation (RAG)

[0048] Retrieval-augmented generation (RAG) is one of the key innovations of this paper. It combines the similarity of historical vulnerabilities with the generative power of large language models to provide higher accuracy and interpretability for CVSS scoring. The specific implementation process is as follows:

[0049] (1) Similar vulnerability retrieval: First, each vulnerability description is converted into a dense vector using text embedding technology based on the BERT model. Dense vectors capture the contextual information of the vulnerability description and can effectively represent the semantics of the text. Then, a retrieval model is used to vectorize all vulnerability descriptions in the historical vulnerability database and calculate the similarity to find the vulnerability record with the most similar semantics to the current vulnerability description. Specifically, a Bi-Encoder model is used for coarse retrieval, and a Cross-Encoder model is used for fine retrieval. This allows us to extract patterns similar to the current vulnerability from historical cases.

[0050] (2) CVSS score generation: After completing the search for similar vulnerabilities, a fine-tuned large language model (such as GPT) is used to generate a CVSS score. The large language model can use its natural language understanding and generation capabilities to integrate the technical characteristics of the vulnerability and information from historical cases to output a CVSS score. The generated score not only reflects the basic impact of the vulnerability, but also adjusts the score based on information about similar vulnerabilities to ensure that the generated result is closer to the actual risk assessment.

[0051] (3) Explainability and transparency: By combining historical data of similar vulnerabilities, the RAG module provides a detailed explanation for each generated CVSS score. Each score is supported by clear historical cases, and users can trace the generation process of the score results. For example, the system will give the specific similarity with historical vulnerabilities and display the generation process of the score through visualization tools. In this way, users can not only obtain the score results, but also trace the logic and basis behind the score, thereby improving the transparency of decision-making. For example, the generated "attack complexity" score may be similar to the "attack complexity" of similar vulnerabilities in history. This traceable process greatly enhances the transparency and credibility of the score.

[0052] Table 1. Effect of the present invention on CVSS severity score prediction

[0053] Indicators / Modules Complete model CWE module ablation RAG module ablation Uncleaned version number Attack Vector 95.12 93.88(-1.24) 90.31(-4.81) 93.94(-1.18) Attack Complexity 98.77 97.91(-0.86) 95.79(-2.98) 96.46(-2.31) Privilege Req 90.34 90.49(+0.15) 82.16(-8.18) 80.60(-9.74) User Interaction 94.08 93.71(-0.37) 88.23(-5.85) 93.18(-0.90) Scope 97.59 94.18(-3.41) 85.92(-11.67) 94.10(-3.49) Confidentiality 89.28 88.92(-0.36) 80.71(-8.57) 86.42(-2.86) Integrity 90.11 89.76(-0.35) 82.65(-7.46) 87.60(-2.51) Availability 89.45 89.12(-0.33) 82.19(-7.26) 87.66(-1.79) CVSS full score 71.45 64.36(-7.09) 40.27(-31.18) 56.20(-15.25)

[0054] Note: The values ​​in brackets are the absolute changes compared with the complete model.

[0055] Appendix 1

[0056]

[0057] Appendix 2

[0058]

Claims

1. A vulnerability severity score prediction method based on a large language model and retrieval enhancement, characterized by: Integrates data preprocessing, CWE hierarchical classification, and retrieval enhancement generation (RAG) technology; includes four stages: data preprocessing, CWE hierarchical classification, retrieval enhancement generation, and score prediction and optimization. Among them: (1) In the data preprocessing stage, redundant information in the vulnerability description is removed to improve the model's ability to understand the vulnerability description and optimize data quality. Specifically: Redundant text in vulnerability descriptions is cleaned and standardized using regular expression algorithms to remove noise, ensuring that the technical features of the vulnerability descriptions are more prominent and reducing the impact of irrelevant information. Lemmatization and stemming techniques are then used to semantically standardize the text to ensure semantic consistency. (2) In the CWE hierarchical classification stage, based on the MITRE View-1003 classification system, a cross encoder is used to jointly optimize the top-level coarse-grained classification and the bottom-level fine-grained classification to accurately predict the CWE category to which the vulnerability belongs, providing more accurate weakness category annotation for the prediction of CVSS scores; (3) Retrieval Enhanced Generation (RAG) phase: The retrieval model uses dense vector retrieval technology to search for vulnerability records similar to the current vulnerability description in the historical vulnerability database, and calculates the semantic similarity between the vulnerability description and the historical records. The large language model is then used to generate the CVSS score, and the generated score is interpreted in combination with historical vulnerability information to ensure the traceability of the scoring process and the transparency of the decision-making. (4) In the scoring prediction and optimization stage, the accuracy of the model's prediction results is evaluated, and the model parameters are adjusted through experimental data to make the scoring results more accurate and improve the model's generalization ability on different data sets.

2. The vulnerability severity score prediction method according to claim 1, characterized in that: The vulnerability description preprocessing described in step (1) cleans the vulnerability description by text noise reduction and standardization, version number detection and replacement, separation of product name and version number, and removal of irrelevant version numbers. In addition, a version number rule set with 14 major categories and 67 subcategories is expanded based on different basic types. These rule sets cover more version number types and further refine various rules.

3. The vulnerability severity score prediction method according to claim 2, characterized in that: The specific process of vulnerability description preprocessing in step (1) is as follows: (1.1) Text denoising and standardization: First, based on the stop word list in the NLP library, high-frequency words that contribute little to semantic understanding are removed to reduce the interference of irrelevant features; All text is converted to lowercase to eliminate word redundancy caused by inconsistent capitalization. Special characters, HTML tags, redundant spaces, and irrelevant punctuation are removed to reduce data noise. Words are converted to their base forms using word form normalization to prevent different forms of the same word from being mistakenly identified as different concepts. (1.2) Version number detection and replacement: Use regular expression matching for the version number format and replace it with a standardized placeholder to avoid interference of the version number information on subsequent semantic modeling; (1.3) Separate product names from version numbers. For some vulnerability descriptions where the version number and product name are closely linked, a dictionary-based splitting method is used to separate the product name and version number, retaining only the product name to ensure that the vulnerability description focuses on the technology. (1.4) Remove irrelevant version numbers; When multiple version numbers appear in a description, normalize them to retain the primary version information and remove redundant enumeration content. (1.5) Based on these basic types, the version number rule set is expanded into 14 major categories, including 67 subcategories.

4. The vulnerability severity score prediction method according to claim 3, characterized in that: The CWE hierarchical classification described in step (2) is achieved by: joint optimization of the top and bottom classifiers, by training a cross encoder to jointly optimize the top coarse-grained classification and the bottom fine-grained classification, thereby enhancing the recognition ability of various features in the vulnerability description and improving the accuracy of CWE prediction; First, MITRE's CWE classification tree is used to separate its top and bottom layers, and a corresponding training dataset is constructed based on the hierarchical information. Then, a "hierarchical classifier" is used to determine whether the input CVE description should belong to the top or bottom layer. Subsequently, within the determined hierarchical range, the "top classifier" and "bottom classifier" are used to perform specific CWE predictions.

5. The vulnerability severity score prediction method according to claim 4, characterized in that: The specific process of step (2) is as follows: (2.1) Hierarchical classifier classification: The input of the hierarchical classifier is the preprocessed CVE description text. After contextual semantic modeling using the Secure-BERT model, the CLS representation of the text is extracted and input into a fully connected classification layer for binary classification. Finally, the model outputs a binary label indicating whether the text belongs to the top or bottom layer. (2.2) Classification of top and bottom classifiers; After completing the hierarchical classification, if the CVE description is predicted to be at the top level, it is sent to the top-level classifier; if it is predicted to be at the bottom level, it is sent to the bottom-level classifier. Both use the Cross-Encoder model to calculate the semantic similarity score of the "CVE description-CWE information" string pair to identify the most likely CWE category. The Cross-Encoder model receives two pieces of text: the CVE description and the CWE text, concatenates them, and feeds them into the BERT model. The Transformer layer captures the contextual associations and semantic dependencies between the two pieces of text. The specific process is as follows: For each CVE description, combine it with all CWE entries corresponding to the top or bottom classifier to generate a string pair. The string pair is input into the Cross-Encoder model, and the semantic similarity score of each pair of texts is calculated, with the similarity score ranging from 0 to 1. The text is sorted in descending order based on the semantic similarity score, and the CWE entry with the highest score is selected as the final prediction result. The prediction results of the Cross-Encoder not only directly output the most likely CWE category, but also provide a complete list of similarity scores.

6. The vulnerability severity score prediction method according to claim 5, characterized in that: The retrieval enhancement generation (RAG) described in step (3) is achieved through the following steps: performing similar vulnerability retrieval, specifically adopting a two-level retrieval scheme of "Bi-Encoder (coarse retrieval) + Cross-Encoder (fine retrieval)"; after completing the similar vulnerability retrieval, obtaining K2 historical vulnerability cases that are most similar to the target vulnerability description; then using a deep learning-based retrieval model to convert the vulnerability description into a dense vector, and searching for the vulnerability case with the most similar semantics to the description in the historical vulnerability database; performing retrieval enhancement generation (RAG) through the large language model (LLM) fine-tuned by Prompt to generate a CVSS score, and further reasoning and correcting the generated result in combination with the retrieved similar vulnerability information to improve the accuracy of the inference of various metric values ​​in the CVSS vector.

Images