Vulnerability detection method of deep learning technology enhanced large language model in operating system kernel scene

Abstract

The invention discloses a vulnerability detection method of a deep learning technology enhanced large language model in an operating system kernel scene, which comprises the following steps: S1, constructing a priori knowledge base including a vulnerability classification hierarchy tree in the operating system kernel scene and a thinking chain basic guidance prompt of a specific vulnerability category; s2, data preparation: using a static tool to generate initial preliminary judgment information, collecting data through open source data collection reserve and operation data feedback, training a deep learning small model after the data size meets the corresponding requirement, and then combining the test output of the small model with the analysis result of the static tool to form final preliminary judgment information; s3, taking the initial judgment information as a query mark, retrieving a corresponding thinking chain basis from a prior knowledge base to guide and prompt to generate a complete target code vulnerability detection thinking chain prompt text, and taking the text as the input of a large model; and S4, analyzing the output of the large model, and generating a vulnerability detection analysis report with high understandability.

Description

Technical Field

[0001] This invention relates to the field of code vulnerability detection, specifically to a vulnerability detection method for a large language model enhanced by deep learning technology in an operating system kernel scenario. Background Technology

[0002] As the core component of an operating system, the kernel is responsible for managing system resources and executing core functions. Its security directly affects the stability and reliability of the entire system. The existence of operating system kernel vulnerabilities can have serious consequences, including but not limited to system crashes, data breaches, malware propagation, system performance degradation, and data corruption. Furthermore, these threats can affect all users of that operating system community, which can be fatal for some open-source operating systems like Linux. Therefore, detecting operating system kernel vulnerabilities is crucial for maintaining system security, availability, and performance.

[0003] However, traditional vulnerability detection methods relying on manual expert review are typically complex and time-consuming, heavily dependent on experts' knowledge of vulnerabilities in specific scenarios, resulting in inconsistent detection effectiveness. Therefore, finding more accurate and efficient automated software vulnerability detection solutions has always been a crucial topic in the field of code vulnerability detection. Existing automated detection solutions can be mainly divided into the following two categories:

[0004] The first category is static code analysis tools, such as FlawFinder, CppCheck, and Rats. These tools typically use pattern matching to detect vulnerabilities and generate corresponding reports. However, these static tools have a high false positive rate and often fail to detect complex vulnerabilities.

[0005] The second category is data-driven methods based on machine learning techniques, such as VulDeePecker, Devign, and SySeVR. These methods typically structure the code to serve as input for deep learning models, thereby achieving binary classification of the detected code. Although these methods have achieved high accuracy on test sets, they are still plagued by problems such as a sharp drop in accuracy in cross-project scenarios, reliance on dataset annotations, and poor interpretability of detection results (containing only 0 or 1).

[0006] Since 2019, from Google T5 to the OpenAI GPT series, large-scale parameter models have made significant progress, especially the ChatGPT large-scale dialogue model, which attracted widespread attention starting at the end of 2022 due to its outstanding performance. Large language model technology has shown potential for application in vulnerability analysis research. However, the application of large language models in the security field currently lacks relevant guidance methods. Inappropriate hints may fail to fully unleash the vulnerability detection capabilities of large language models, and may even result in lower classification performance than random classifiers.

[0007] Therefore, it is necessary to propose a vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing methods by providing a suggestion construction method that embeds static tools and deep learning small models. By combining the results of both and generating a corresponding thought chain based on the operating system kernel vulnerability classification hierarchy tree in the prior knowledge base, appropriate suggestions can be formed to significantly improve the vulnerability classification performance of large language models, generate highly understandable code vulnerability analysis reports, and complete the vulnerability detection task of the operating system kernel.

[0009] like Figure 1 To address the aforementioned technical problems, this invention provides the following technical solution: A vulnerability detection method for large language models enhanced by deep learning technology in an operating system kernel scenario. The method comprises the following steps:

[0010] S1. Construct a priori knowledge base, including a vulnerability classification hierarchy tree in the operating system kernel scenario and basic guidance prompts for the mind chain corresponding to a single vulnerability category;

[0011] S2. Data preparation: Use static tools to generate initial judgment information. Data can be collected through open-source data collection and storage and running data feedback. After the amount of data reaches the corresponding requirements, train a small deep learning model. Then combine the test output of the small model with the analysis results of static tools to form the final initial judgment information.

[0012] S3. Use the initial judgment information as a query tag, construct the target code vulnerability detection mind chain prompt based on the corresponding mind chain basic guidance prompt in the prior knowledge base, and use it as the input of the large model;

[0013] S4. Parse the output of the large model to generate a highly understandable vulnerability detection and analysis report;

[0014] Preferably, the vulnerability classification hierarchy tree in the operating system kernel scenario in S1 mainly includes:

[0015] Vulnerabilities are first categorized into six basic types: errors caused by incomplete security features, errors caused by program execution logic, errors caused by memory-related resources, errors caused by numerical calculations, errors caused by failure to standardize (validate, restrict) exchanged data, and unknown errors.

[0016] Based on the six base class divisions, and using the classification criteria provided by CWE (Common Weakness Enumeration), each base class is further refined, ultimately forming... Figure 7 The diagram shows a hierarchical classification tree with 45 leaf nodes and a depth of three levels. This tree is designed to fully consider operating system kernel scenarios, making it more targeted. Six base classes cover all possible vulnerability categories that may occur in the operating system kernel. Furthermore, the classification tree is pruned to remove categories less common in kernel scenarios, such as SQL injection, while highlighting categories with higher frequency of occurrence.

[0017] Preferably, the basic guidance prompts for the thought chain corresponding to a single vulnerability category in S1 are generated following the paradigm:

[0018] Semantics: Understanding the function of the code.

[0019] Logic: Analyze the structure of the code.

[0020] Internal risks: Identify components that may introduce vulnerabilities.

[0021] External risks: Inspect insecure features that could lead to vulnerabilities.

[0022] Generating a thought chain: Integrate the information obtained above to generate a thought chain, progressively checking for potential vulnerabilities. While adhering to a unified paradigm, the basic guidance prompts for the thought chain also consider the specificities of single-node vulnerability categories, supplementing and expanding the original basic guidance prompts. For example, the basic guidance prompts for null pointer exception errors include supplementary content on pointer identification, location, allocation, and release. Furthermore, the basic guidance prompts for parent and child nodes in the classification hierarchy tree also have an inheritance property; the basic guidance prompts for child nodes are 4-5 additional points based on those for parent nodes.

[0023] Preferably, the method for generating the initial judgment information in S2 includes:

[0024] The first component consists of static tool analysis results, using tools such as FlawFinder, CppCheck, and Rats. These tools analyze the input code snippets and generate detection reports. Based on the report results, they are mapped to a vulnerability classification hierarchy tree, generating the first component of the initial assessment information. The selected static tools fully consider the characteristic that operating system kernel scenarios are mostly written in C language, demonstrating good performance in detecting and identifying C language functions. More static tools can be integrated in the future based on data interfaces.

[0025] The second component is the small model test results. Since the small model needs to be trained with labeled sample data, the data preparation is first performed, i.e., the detection task is performed without the output of the small model. Then, the detection results are used as the training set to train the deep learning small model, and the trained model is used to test the subsequent input code. The confidence scores in the test results are added to the initial judgment information data structure.

[0026] Preferably, the selection of the deep learning small model in S2 includes:

[0027] This invention provides several small deep learning models as options, including DNN, LSTM, GRU, BiLSTM, BiGRU, and TextCNN. Generally, BiLSTM yields the best vulnerability detection results, presumably because its model structure is closest to the Transformer-based structure of large language models.

[0028] Preferably, the specific details of generating the target code vulnerability detection thought chain in S3 include:

[0029] Based on the mapping from the first component of the initial judgment information in S2, namely the static tool analysis results, to the operating system kernel classification hierarchy tree, the corresponding basic guidance prompts of the thinking chain are extracted. Then, based on the second component of the initial judgment information, namely the test results of the deep learning small model, the strictness of the prompt template is adjusted. Combining the content of the two and generating high-quality input that stimulates the classification potential of the large language model with strict output format requirements.

[0030] Preferably, the specific details of parsing the large model output in S4 to generate a highly understandable vulnerability detection and analysis report include:

[0031] Because S3 restricts the output format of large language models, the output can be uniformly processed using string parsing methods to generate corresponding vulnerability detection reports according to S3's format requirements. The specific content of the detection report is determined by the output format and may include: whether a vulnerability exists, the impact of the vulnerability, and the cause of the vulnerability. The prompt templates in S3 can be adjusted to generate more personalized detection reports. Occasionally, the large language model may produce ambiguous judgments regarding the existence of vulnerabilities; these results require a secondary assessment combining the initial judgment information and the large language model's own output.

[0032] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: Based on a large language model, this invention realizes vulnerability detection and result analysis in the operating system kernel scenario. Specifically, it utilizes the inherent vulnerability classification potential of the large language model to detect input kernel code and generate a highly understandable vulnerability detection and analysis report. Through static tool analysis results, it generates a corresponding category of vulnerability thought chain and embeds a deep learning mini-model to allow the large language model to better understand the feature information of the corresponding project scenario in context learning, thereby realizing implicit fine-tuning of the large language model and stimulating its vulnerability classification potential. In comparison, this invention achieves a significantly higher accuracy rate in vulnerability detection than traditional static tools and deep learning-based vulnerability detection tools. It also solves the problem of high false positive rates in static tools, allowing engineering teams to focus more on genuine security threats and greatly improving overall work efficiency. Through the construction of high-quality suggestions, its accuracy far surpasses other vulnerability detection methods based on large language models. By embedding deep learning for implicit fine-tuning rather than explicitly fine-tuning the open-source large language model, it significantly reduces computational power consumption and resource requirements while improving accuracy. Compared to the results produced by deep learning vulnerability detection tools (which only use 0 and 1 to represent the presence or absence of vulnerabilities), this invention can generate detailed and clearly targeted vulnerability detection reports, which can be used for subsequent work such as vulnerability localization, impact assessment, and remediation. Attached Figure Description

[0033] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0034] Figure 1 This is a general flowchart of the present invention;

[0035] Figure 2 This is a detailed flowchart of the present invention;

[0036] Figure 3 This is a flowchart of step S1 of the present invention;

[0037] Figure 4 This is a flowchart of step S2 of the present invention;

[0038] Figure 5 This is a flowchart of step S3 of the present invention;

[0039] Figure 6 This is a sample vulnerability detection report from the implementation example;

[0040] Figure 7 This is a schematic diagram illustrating errors caused by incomplete security features in the operating system kernel vulnerability classification hierarchy tree of this invention.

[0041] Figure 8 This is a schematic diagram illustrating the errors caused by the execution logic of the operating system kernel vulnerability classification hierarchy tree program in this invention.

[0042] Figure 9 This is a schematic diagram illustrating errors caused by memory-related resources in the hierarchical classification tree of operating system kernel vulnerabilities in this invention.

[0043] Figure 10 This is a schematic diagram illustrating the error caused by numerical calculation of the operating system kernel vulnerability classification hierarchy tree in this invention.

[0044] Figure 11 This is a schematic diagram illustrating the error caused by the lack of standardization (verification, restriction) of exchanged data in the operating system kernel vulnerability classification hierarchy tree of this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The term "performance" used in this article refers to the method's performance in terms of accuracy, including precision, recall, F1 score, and accuracy across all samples.

[0047] The term "precision" used in this article refers to the percentage of samples classified as positive that are actually positive. It can be calculated using the following formula: Precision = Number of true positives / (Number of true positives + Number of false positives).

[0048] The term "recall" used in this article refers to the percentage of all true positive samples that are correctly classified as positive. It can be calculated using the following formula: Recall = Number of true positive samples / (Number of true positive samples + Number of false negative samples).

[0049] The term "F1 score" used in this paper refers to the weighted harmonic mean of precision and recall, used to comprehensively evaluate the model's performance. It can be calculated using the following formula: F1 score = 2 * (precision * recall) / (precision + recall). The term "precision" used in this paper refers to the ratio between the number of correctly classified samples and the total number of samples. It can be calculated using the following formula: Precision = (Number of true positives + Number of true negatives) / Total number of samples.

[0050] This invention provides an algorithm, including the following steps:

[0051] S1. Construct a priori knowledge base, including a vulnerability classification hierarchy tree in the operating system kernel scenario and basic guidance prompts on the mind chain of specific vulnerability categories;

[0052] S2. Data preparation: Use static tools to generate initial judgment information. Data can be collected through open-source data collection and storage and running data feedback. After the amount of data reaches the corresponding requirements, train a small deep learning model. Then combine the test output of the small model with the analysis results of static tools to form the final initial judgment information.

[0053] S3. Using the initial judgment information as a query tag, retrieve the corresponding basic guidance prompts of the thinking chain from the prior knowledge base to generate a complete target code vulnerability detection thinking chain prompt text, and use this as the input of the large model;

[0054] S4. Parse the output of the large model to generate a highly understandable vulnerability detection and analysis report. The flowchart of S1 is as follows: Figure 3 As shown, it includes the following steps:

[0055] S1-1: Identify the six basic categories of errors in the operating system kernel scenario: errors caused by incomplete security features, errors caused by program execution logic, errors caused by memory-related resources, errors caused by numerical calculations, errors caused by failure to standardize (verify, restrict) exchanged data, and unknown errors.

[0056] S1-2: Considering the specific characteristics of the operating system kernel scenario, each single base class is refined according to the classification criteria provided by CWE (Common WeaknessEnumeration), forming a vulnerability classification hierarchy tree with three levels of depth and 45 leaf nodes in the operating system kernel scenario. By performing more granular classification, vulnerability categories can be determined more accurately, while also better fitting the special properties of the operating system kernel, making this invention more professional in the kernel scenario.

[0057] S1-3: Determine the paradigm to be followed in the basic guidance of the thought chain corresponding to a single vulnerability category. This paradigm must include semantics, logic, internal risks, external risks, and thought chain integration and generation. This part of the paradigm reports the general steps of vulnerability detection to ensure the generality of vulnerability detection and to deal with special cases where no category reference is given in the initial judgment information.

[0058] S1-4: Taking into account the special characteristics of single-node vulnerability categories, and while adhering to a unified paradigm, the original basic guidance prompts of the thinking chain are supplemented and expanded, so that the thinking chain can more accurately guide the large language model to the corresponding problem domain.

[0059] S1-5: Considering the parent-child relationship of nodes in the vulnerability classification hierarchy tree under the operating system kernel scenario, 4-5 additional basic guidance prompts are made on the basis of the parent node's basic guidance prompts as the basic guidance prompts of the child node's basic guidance prompts, so that the basic guidance prompts of the parent and child nodes' basic guidance prompts also have the property of inheritance.

[0060] The flowchart of S2 is as follows Figure 4 As shown, it includes the following steps:

[0061] S2-1: Integrates static tools such as FlawFinder, CppCheck, and Rats.

[0062] S2-2: Map the analysis results of the input code by the static tool combination into nodes of the vulnerability classification hierarchy tree in the operating system kernel scenario, and use it as the first component of the initial judgment information.

[0063] S2-3: Data preparation. In this process, only the first component is used as preliminary information to pass into subsequent processes, and the labels generated by the samples during this stage are collected.

[0064] S2-4: After the number of collected sample labels reaches the target number, divide them into training and validation sets in an 8:2 ratio, select hyperparameters, start training the deep learning small model, and save the best training results.

[0065] S2-5: Use the best small model training results to test subsequent input samples, and use the confidence level in the test results as the second component of the initial judgment information.

[0066] The flowchart of S3 is as follows: Figure 5 As shown, it includes the following steps:

[0067] S3-1: Based on the first component of the initial assessment information—the mapping from the analysis results of the static tool combination to the nodes of the vulnerability classification hierarchy tree in the operating system kernel scenario—retriev the basic guidance prompts for the corresponding vulnerability category from the prior knowledge base. S3-2: Based on the second component of the initial assessment information—the confidence level of the deep learning small model's test results on the input samples—adjust the strictness of the prompt template accordingly. Skip this step during data preparation and proceed to subsequent steps.

[0068] S3-3: Combine the above two points and generate the final input prompts for the large language model according to strict output format requirements. The output requirements must include: whether the vulnerability exists, the impact of the vulnerability, and the reason for the vulnerability.

[0069] S3-4: Pass this input suggestion to the actual selected large language model.

[0070] S4 includes the following steps:

[0071] S4-1: Obtain the output of the large language model.

[0072] S4-2: Parse the output of the large language model according to the designed output format requirements, extracting three parts: whether a vulnerability exists, the impact of the vulnerability, and the cause of the vulnerability, to create a final, highly understandable vulnerability analysis report. Based on the information provided in the vulnerability analysis report, subsequent processing operations such as vulnerability location and remediation can be performed. Example: The method of this invention is implemented on an existing real-world dataset.

[0073] The basic datasets used in this embodiment come from the Linux and Debian open-source operating system kernels. 37,484 datasets were collected from Linux and 14,935 from Debian, with functions as the sample granularity. The data is uniformly saved in JSON format. Next, the hyperparameters for training the small deep learning model are selected. The hyperparameters used in this embodiment include: BiLSTM as the training model, Word2Vec as the word embedding method, 100 as the embedding vector size, 50 training epochs, and stochastic gradient descent (SGD) as the optimizer. These choices represent the hyperparameter combinations that generally perform best for the model. Adjustments to these hyperparameters can be attempted without considering cost.

[0074] Next, the large language model used in this implementation case is selected. This implementation case uses the GPT3.5-turbo-0613 large language model. Then, the steps of this invention are run with two datasets as input, and the log information during the process is saved, resulting in two summary analysis reports. Statistical analysis is then performed on the results. The metrics used include precision, recall, F1 score for both vulnerable and non-vulnerable samples, and accuracy for all samples. The results are shown in Table 1.

[0075] Table 1 shows the experimental results of the method on Linux and Debian datasets:

[0076]

[0077] Next, the existing vulnerability detection methods Devign and SySeVR were run on Linux and Debian datasets, and the results were statistically analyzed. A common vulnerability detection method performance evaluation system was adopted, with specific metrics including recall rate of vulnerability samples, F1 score, and precision of all samples. The comparison results are shown in Table 2.

[0078] Table 2 Comparison of Results

[0079]

[0080] Table 2 clearly shows that the vulnerability detection performance of this method is superior to Devign and SySeVR, and it is of great significance in vulnerability detection of operating system kernel code. Example of vulnerability detection report results in the embodiments is shown below. Figure 6 As shown, it is clear that the vulnerability detection report obtained by implementing this invention has a clear logic, comprehensive analysis, and high comprehensibility. It is easy to locate the vulnerability through the report content and carry out subsequent work such as vulnerability repair.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0082] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0083] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario, characterized by: The detection method includes the following steps: S1. Construct a priori knowledge base, including a vulnerability classification hierarchy tree in the operating system kernel scenario and basic guidance prompts for the corresponding vulnerability categories' thought chains; S2. Data preparation: Use integrated static tools to obtain preliminary information for subsequent steps. After the amount of data reaches the required level, train a small deep learning model. Then, combine the test output of the small model with the analysis results of the static tools to form the final preliminary information. S3. Use the initial judgment information as a query tag, construct the target code vulnerability detection mind chain prompt based on the corresponding mind chain basic guidance prompt in the prior knowledge base, and use it as the input of the large model; S4. Parse the output of the large model to generate a highly understandable vulnerability detection and analysis report.

2. The vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario according to claim 1, characterized in that: S1 includes the following steps: S1-1: Identify the six basic categories of errors in the operating system kernel scenario, namely: errors caused by incomplete security features, errors caused by program execution logic, errors caused by memory-related resources, errors caused by numerical calculations, errors caused by failure to standardize (verify, restrict) exchanged data, and unknown errors. S1-2: Considering the specific characteristics of the operating system kernel scenario, each single base class is refined according to the classification criteria provided by CWE (Common Weakness Enumeration), forming a vulnerability classification hierarchy tree with three levels of depth and 45 leaf nodes for the operating system kernel scenario. S1-3: Determine the paradigm to be followed by the basic guidance prompts for the thought chain corresponding to a single vulnerability category, which must include semantics, logic, internal risks, external risks, and thought chain integration and generation. S1-4: Taking into account the special characteristics of single-node vulnerability categories, and while adhering to a unified paradigm, the original basic guidance and tips for the thought chain are supplemented and expanded. S1-5: Considering the parent-child relationship of nodes in the vulnerability classification hierarchy tree under the operating system kernel scenario, 4-5 additional basic guidance prompts are made on the basis of the parent node's basic guidance prompts as the basic guidance prompts of the child node's basic guidance prompts, so that the basic guidance prompts of the parent and child nodes' basic guidance prompts also have the property of inheritance.

3. The vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario according to claim 1, characterized in that, S2 includes the following steps: S2-1: Integrates FlawFinder, CppCheck, and Rats static tools S2-2: Map the analysis results of the input code using static tools to transform them into nodes of a vulnerability classification hierarchy tree in the operating system kernel scenario, and use this as the first component of the initial judgment information. S2-3: Data preparation. During this process, only the first component is used as initial information for subsequent steps, and the labels generated from samples during this stage are collected. S2-4: After collecting the target number of sample labels, divide them into training and validation sets in an 8:2 ratio. After determining the selection of each hyperparameter, start training the deep learning mini-model and save the best training results. S2-5: Use the best small model training results to test subsequent input samples, and use the confidence level in the test results as the second component of the initial judgment information.

4. The vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario according to claim 1, characterized in that, S3 includes the following steps: S3-1: Based on the first component of the initial judgment information, namely the mapping from the analysis results of the static tool combination to the nodes of the vulnerability classification hierarchy tree in the operating system kernel scenario, retrieve the basic guidance prompts of the corresponding vulnerability category from the prior knowledge base; S3-2: Based on the second component of the initial judgment information, namely the confidence level in the test results of the deep learning small model on the input sample, adjust the strictness of the prompt template accordingly. Skip this step during data preparation and proceed to subsequent steps; S3-3: Combine the above two contents and generate the final input prompts for the large language model according to strict output format requirements. The output requirements must include: whether the vulnerability exists, the impact components of the vulnerability, and the cause of the vulnerability; S3-4: Input this target code vulnerability detection mind chain prompt text into the actually selected large language model.

5. The vulnerability detection method for large language models enhanced by deep learning technology in the operating system kernel scenario according to claim 1, characterized in that, S4 includes the following steps: S4-1: Obtain the output result of the large language model; S4-2: Parse the output result of the large language model according to the designed output format requirements, and extract three parts: whether the vulnerability exists, the impact of the vulnerability, and the cause of the vulnerability, as the final vulnerability analysis report with high comprehensibility.

Images