Large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion
By combining data enhancement with LoRA, we construct a multi-round prompt chain and a dynamic fusion controller, which solves the performance deficiencies of Code LLMs in vulnerability detection and explanation, achieves accurate identification and detailed explanation of vulnerabilities, and improves the vulnerability detection capability of the model.
Patent Information
- Application Number
- CN202510671020.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-09-26
AI Technical Summary
Existing Code LLMs have insufficient performance in vulnerability detection tasks, especially in terms of generalization and context understanding capabilities in professional code fields, and there is still a gap in research on vulnerability detection and explanation.
By adopting the method of data enhancement and LoRA fusion, we construct five enhanced vulnerability code datasets with different feature distributions by automatically parsing the vulnerability code differences. We also use LoRA technology to fine-tune the pre-trained Qwen2.5-Coder-7B base model to generate an independent LoRA adaptation layer. Combined with multi-round prompt chains and dynamic fusion controllers, we achieve vulnerability detection and interpretation.
The accuracy and efficiency of Code LLMs in vulnerability detection and explanation are improved. It can accurately identify vulnerable lines and their context-related lines, generate detailed vulnerability explanations, and enhance the performance and adaptability of the model in the field of vulnerability detection.
Smart Images
Figure CN120705871A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to fields such as source code vulnerability detection technology, and specifically, to a large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion. Background Art
[0002] Software vulnerabilities are a significant security issue in cyberspace security, and code vulnerabilities are the foundation of software vulnerabilities. The Open Source Security and Risk Analysis (OSSRA) organization has found that the number of code vulnerabilities has increased annually in recent years, while manual vulnerability detection is difficult and time-consuming. Therefore, effective, large-scale code vulnerability detection methods are urgently needed.
[0003] In recent years, with the application and development of deep learning technology, researchers have proposed deep learning-based code vulnerability detection methods. These methods, such as Sysevr (Sysevr: A framework for using deep learning to detect software vulnerabilities) and DeepWukong (DeepWukong: Statically detecting software vulnerabilities using a deep graph neural network), treat code vulnerability detection as a binary classification problem. By converting code into various abstract forms, such as strings, trees, or graphs, they calculate the similarity between code snippets to determine whether the target code contains vulnerabilities. However, these methods all treat code snippets as text, differing only in the way the text is embedded. Whether they can truly reflect the logic and structure of the code remains to be further verified.
[0004] With the introduction of the Transformer architecture, many Transformer-based large language models (LLMs), such as CodeBERT (2020), GraphCodeBERT (2021), and UnixCoder (2021), have made significant progress in vulnerability detection tasks and demonstrated excellent performance in multiple code-related downstream tasks. With the rapid development of decoder-only LLMs, code-specific language models (LLMs) have attracted widespread attention in both academia and industry. These models are typically further trained on general-purpose pre-trained LLMs to adapt to the syntax and semantics of programming languages, as well as code reasoning tasks. In recent years, several outstanding code LLMs have emerged, such as the StarCoder series (2023), CodeLlama series (2023), DeepSeekCoder series (2024), and CodeQwen1.5 (2024). These models have demonstrated excellent performance in tasks such as code generation, code completion, bug fixing, and code translation, and have achieved remarkable results on multiple code-related benchmarks. However, despite the continued progress of CodeLLMs in both open source and proprietary domains, there remains a gap compared to the state-of-the-art large-scale proprietary LLMs. For example, top closed-source models such as Claude-3.5-Sonnet (2024) and GPT-4o (2024) demonstrate superior capabilities across multiple dimensions, including code understanding, generation, optimization, and reasoning. These models, leveraging larger data sets for training, enhanced reasoning capabilities, and more efficient architectural optimizations, demonstrate the potential to surpass existing CodeLLMs in code-related tasks. Therefore, further enhancing the capabilities of CodeLLMs to achieve greater generalization and contextual understanding in specialized code domains remains a key challenge.
[0005] Currently, the performance of CodeLLMs for security-oriented tasks, particularly vulnerability detection, remains largely unexplored. Some research has applied LLMs to software engineering, such as automated program maintenance. However, these studies primarily focus on using LLMs for generation-based tasks, leaving a significant gap in the application of LLMs to classification tasks. Summary of the Invention
[0006] The purpose of the present invention is to address the deficiencies of the prior art and provide a large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion, which effectively extracts the vulnerability line feature information, semantic information and text information of the code, and can help developers quickly verify the authenticity of the vulnerability and obtain vulnerability solutions.
[0007] The present invention is implemented through the following technical solution: a large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion, including the following specific steps: 1) Use scripts to collect the vulnerability code field (func_before), repair information field (patch_diff), repaired code field (func_before), vulnerability explanation information field (cve_description), and common defect enumeration type field (cwe_type) in the source code vulnerability detection dataset; 2) Automatically parse and extract vulnerable lines (vulnerable_lines) based on the differences between the code before and after the vulnerability fix (i.e., the func_before and func_after fields); 3) Based on the vulnerability code information obtained in steps 1) and 2), the thought chain reasoning method is used to guide the large model to generate vulnerability explanation information with deeper semantics, enhancing the large model's ability to understand the vulnerability causes and repair methods; 4) The vulnerability code and fix information obtained in step 1), the vulnerability line information obtained in step 2), and the vulnerability explanation information obtained in step 3) are processed separately to construct five enhanced vulnerability code datasets with different feature distributions; 5) Based on the obtained enhanced vulnerability code dataset, the pre-trained Qwen2.5-Coder-7B base model is trained using the parameter efficient fine-tuning technology (LoRA) to generate an independent LoRA adaptation layer to learn different task features such as vulnerability detection and vulnerability explanation; 6) After obtaining the corresponding LoRA adaptation layer for each data set, load the LoRA adaptation layer corresponding to the vulnerability detection to complete the vulnerability detection task; initialize the weight coefficient, use the weight coefficient to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficient using the gradient-free update algorithm to complete the vulnerability explanation task.
[0008] In order to further better implement the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: Step 1) is specifically: for each source code vulnerability detection dataset (such as Big-Vul, Prime-Vul, etc.), a python script is used to automatically extract the vulnerability code field (func_before), repair information field (patch_diff), repaired code field (func_before), vulnerability explanation information field (cve_description) and common defect enumeration type field (cwe_type) in the source code vulnerability detection dataset, and save the extracted various information in json format; when extracting, note that the function itself needs to be filtered; the filtering operation includes: deleting incomplete functions, including functions that do not end with} and functions that end with ;; excluding functions that have not been actually modified (no lines are added or deleted); excluding functions that have been modified by more than 70% (this is because the original function and the modified function are no longer strongly related in semantics and grammar due to too many modifications); excluding functions with less than 5 lines.
[0009] In order to further better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: the step 2) includes the following specific steps: 2.1) Differential Extraction: Automatically analyze the differences between the vulnerable code and the fixed code using the Python difflib library, recording added and deleted fields and their corresponding line numbers. 2.2) Vulnerability Line Determination: For deleted lines, they are directly marked as vulnerable lines. For lines added in the repaired code, the lines directly associated with the context of adding the line and appearing in the vulnerable code are marked as vulnerable lines. 2.3) Determining Directly Related Lines in Context: Use the source code analysis tool joern to parse the repaired code and generate the corresponding node.csv and egde.csv files for each C / C++ file. In the node.csv file, use regular expression matching to find the target added line. Then, return to egde.csv and use the CONTROL field to determine the lines directly related to the target line. Finally, use the joern tool to parse the vulnerable code and perform regular expression matching on the directly related lines found in the repaired code to locate the corresponding vulnerable line.
[0010] In order to further better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: the step 3) includes the following specific steps: 3.1) Sequentially extract the vulnerability code field, repair information field, repaired code field, vulnerability explanation information field, general defect enumeration type field, and the code vulnerability line information located in step 2) obtained in step 1), and call the API design prompt word of the large model DeepSeek; The prompt words are as follows: I want you to act as a vulnerability detection expert . This is acodesnippet:{func_before}.lets think step by step. Firstly, what is the purpose of the function in the following code snippet. Please summarize the answer in onesentence. The code snippet above has a vulnerability type : {cwe_id} and thevulnerabilitydescription is :{cve_description}. Then according to thevulnerable description, please only output the vulnerable lines of the codesnippet. The vulnerable lines are {vulnerability_lines} and the correct way to fix it is by {patch_diff}. Now please double-check your answer and only output the repaired code snippet according to the patch diff. The code snippet after refinement is {func_after}.Now pleasereconsider your answer andoutput the vulnerability interpretation based on the above information. DO NOT INCLUDE ANYTHING ELSE EXCEPT VNLNERABILITYINTERPRETATION IN YOUR RESPONSE. 3.2) Enable the DeepSeek historical message mechanism, so that the domestic large-scale DeepSeek model can integrate the information of previous rounds of prompt words to obtain a complete chain of thought, and finally output more detailed vulnerability explanation information including the cause of the vulnerability triggering and repair information.
[0011] In order to further better realize the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: in the step 4), the five enhanced vulnerability code data sets with different feature distributions constructed are: vulnerability detection (vulnerability detection) data set, vulnerability type detection (vulnerability type) data set, vulnerability line location (vulnerable lines location) data set, vulnerability repair (vulnerability patch) data set, vulnerability interpretation (vulnerability interpretation) data set; wherein, the input (input) of each enhanced vulnerability code data set is the vulnerability code, and the output is the corresponding task objective.
[0012] In order to further better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: the step 5) is specifically: Use LoRA technology to fine-tune the five enhanced vulnerability code datasets generated in step 4) for the pre-trained Qwen2.5-Coder-7B base model to generate five independent LoRA adaptation layers; during the fine-tuning process, the input data is Instruction and Input, and the output data is the label corresponding to each enhanced vulnerability code dataset task (wherein, the label of the vulnerability detection dataset is 0 / 1 binary classification, representing the presence or absence of a vulnerability; the label of the vulnerability type detection dataset is the general defect enumeration type; the label of the vulnerability line location dataset is the code vulnerability line; the label of the vulnerability repair dataset is the code repair information; the label of the vulnerability explanation dataset is the enhanced vulnerability explanation information obtained in step 3)). Among them, the Input is the vulnerability code to be detected on the five enhanced vulnerability code datasets, and the Instruction is constructed on each enhanced vulnerability code dataset through multiple rounds of prompt chains. The following example: Vulnerability detection: Detect whether the following code is vulnerable or not. Onlyoutput '1'(vulnerable) or '0'(non-vulnerable) Vulnerability type detection: The following code is vulnerable. Please only output itsvulnerability type with the format: CWE-416 Vulnerability explanation: The following code is vulnerable and its vulnerability typeis CWE-476. Please output its vulnerability interpretation in one sentence. Vulnerable line location: The following vulnerable code with a vulnerable type CWE-77 has a vulnerability interpretation: Before picocombefore 2.0, there is a command injection vulnerability in the'send and receive file' command because the command line is executed by / bin / sh unsafely. Please output its vulnerable lines with the following format: {'line_number' : 'Vulnerable line'} Vulnerability fix: The following vulnerable code with a vulnerable type has a vulnerability interpretation: Multiple unspecified vulnerabilities in Google Chrome before 36.0.1985.143 allow attackers to cause a denial of service or possibly have other impact via unknown vectors.. Its vulnerable lines are {'8':'return narrowPrecisionToFloat(sqrt((pow(ctm.xScale(), 2)+ pow(ctm.yScale(), 2)) / 2));'}. Please only output the lines needed to be added or deleted to make the code correct with the following format: {'add':[lines to be added], 'deleted':[lines to be deleted]}"; Hyperparameter selection: During fine-tuning, considering performance factors, this invention also uniformly padded / truncated all input data to a length of 4096 tokens. Different LoRA rank values (LoRA rank (r)) were set for different tasks: the LoRA rank for the vulnerability detection task was set to 8, and the LoRA rank for other generation tasks was set to 16. In other words, the LoRA rank (r) set in this invention is different: for the vulnerability detection task, r = 8, and for the other generation tasks, r = 16.
[0013] In order to further better implement the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, in order to achieve multi-task fusion capability, after training for different tasks respectively, in the vulnerability interpretation task, the LoRA adaptation layers of multiple tasks are integrated into a unified output path in a weighted manner by constructing a dynamic fusion controller, thereby improving the interpretation effect. In particular, the following setting method is adopted: in the step 6), when completing the vulnerability detection task, the vulnerability detection LoRA adaptation layer is loaded using the Python transformers library to complete the vulnerability detection task; Initialize the weight coefficients, use the weight coefficients to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficients using a gradient-free update algorithm to complete the vulnerability explanation task. The specific steps include the following: 6.2.1) Weight coefficient initialization: Use the cosine similarity of the task to initialize the weight of each LoRA adaptation layer, and then build a parameterized dynamic fusion controller. The specific formula is as follows: ; in, That is, the learnable fusion parameters can be dynamically adjusted through mechanisms or algorithms; That is, the original parameters of the model; Parameters of each LoRA module; 6.2.2) Gradient-Free Optimization Algorithm: The Cross Entropy Method (CEM) algorithm is used for coefficient optimization. CEM is a stochastic optimization algorithm that approaches the optimal solution by sampling from a probability distribution, evaluating the sample quality, and updating the distribution parameters. The evaluation criterion (loss function) for vulnerability explanation quality is defined as follows: ; Among them, IOU (Intersection over Union, positioning accuracy) is used to measure the overlap between the predicted area and the real area, and BLEU-4 (Bilingual Evaluation Understudy) is used to measure the matching degree between the text generated by the model and the reference text (ground truth). In this invention, A and B are set to 0.5 and 0.5 respectively; 6.2.3) Fusion coefficient update: Using the CEM algorithm constructed in 6.2.2), sample and evaluate a small number of vulnerability explanation samples (1% in this paper) to update the fusion parameters. , realizing the dynamic fusion of different LoRA modules and finally outputting the vulnerability explanation; achieving the goal of enabling the pre-trained Qwen2.5-Coder-7B base model to load the LoRA adaptation layer parameters fine-tuned by different data sets, so that the pre-trained Qwen2.5-Coder-7B base model can obtain knowledge from different fields, effectively improving the performance and adaptability of the model in the field of vulnerability explanation.
[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The study found that the vulnerability detection efficacy of CodeLLMs depends largely on the quality of the prompts provided to the model (task descriptions and other relevant information). Therefore, to prevent CodeLLMs from learning false information during training, this paper proposes a data augmentation method that enables CodeLLMs to perform effective multi-task learning on a single dataset, so as to learn the deep characteristics of the vulnerability itself rather than the superficial text information. At the same time, due to the excellent performance of LLM in text generation tasks, this paper not only focuses on the binary classification task of large models, but also proposes an efficient method for generating vulnerability explanations to help developers effectively identify and locate vulnerabilities.
[0015] (2) The present invention combines difflib difference analysis and Joern code parsing technology to accurately identify the vulnerable lines and their context-related lines, and accurately locate the specific vulnerable lines in the vulnerable code.
[0016] (3) This invention designs a multi-round prompting chain (Step-by-Step Prompting), leveraging the reasoning capabilities of the DeepSeek large model to generate detailed explanations that include vulnerability causes and repair logic, overcoming the shortcomings of traditional tools in terms of insufficient explanation capabilities. By integrating contextual information through a historical message mechanism, the invention outputs structured explanation results (such as trigger causes and repair suggestions), making the vulnerability explanation generation effect public and transparent.
[0017] (4) This paper innovatively proposes an enhanced vulnerability code dataset for five major tasks: vulnerability detection, vulnerability type classification, vulnerability line location, vulnerability repair, and vulnerability explanation. At the same time, it proposes a dynamic fusion controller, which initializes weights based on cosine similarity and optimizes the fusion parameters (∂i) through the CEM algorithm to achieve adaptive combination of multi-domain knowledge. The performance of the model output vulnerability explanation is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a flowchart of a large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion proposed in a specific embodiment of the present invention. DETAILED DESCRIPTION
[0019] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.
[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.
[0022] Glossary: Qwen2.5-Coder-7B base model: Qwen2.5Coder-7B is a large open source code-specific model launched by the Alibaba Cloud Tongyi Qianwen team in September 2024. It has 7 billion parameters and is optimized for code generation, understanding, and debugging tasks. This paper uses its 8-bit quantized version to reduce resource consumption while ensuring training results.
[0023] 8-bit quantization: A technique that reduces the precision of values in neural networks to reduce model computation and storage overhead. It compresses model parameters (such as weights) and activation values from traditional 32-bit floating-point numbers (FP32) to 8-bit integers (INT8), significantly improving computational efficiency and reducing resource consumption.
[0024] LoRA (Low-Rank Adaptation): A highly efficient parameter fine-tuning method designed specifically for large pre-trained language models. Its core idea is to approximate model parameter updates through low-rank matrix factorization, significantly reducing the number of parameters and computational resources required for training while maintaining model performance. During fine-tuning, LoRA primarily operates on the attention and fully connected layers of the model's Transformer blocks. Trainable low-rank matrices A and B are then added alongside the selected layers, freezing the original weights. Since only the low-rank matrices are optimized during training, the number of paths for gradient propagation is reduced, saving graphics memory. After training, ΔW = A·B can be optionally superimposed on the original weights to merge them into a single model. ΔW here refers to the LoRA adaptation layer mentioned earlier. Five such adaptation layers were trained for the task, each incorporating domain-specific knowledge from the corresponding dataset.
[0025] Joern: An open source tool that can generate corresponding CPG for source code.
[0026] The CEM algorithm is a stochastic optimization algorithm that approaches the optimal solution by sampling from a probability distribution, evaluating the quality of the samples, and updating the distribution parameters. The core idea of the CEM algorithm is as follows: First, the solution space is represented by a probability distribution (such as a Gaussian distribution) and multiple candidate solutions are sampled from it. The quality of these solutions is then evaluated based on an objective loss function. The optimal partial solution is selected and the parameters of the probability distribution are updated to make the new distribution more likely to generate high-quality solutions. Finally, the above steps are repeated until convergence or the maximum number of iterations is reached.
[0027] Multi-round Prompt Chain: Developed from the Chain of Thought (CoT), the core concept of CoT is to require the model to explicitly demonstrate its reasoning process (such as mathematical calculation steps and logical reasoning) before generating a final answer. This mimics human progressive thinking and requires the model to demonstrate complete reasoning within a single response. Multi-round Prompt Chain, on the other hand, uses a series of ordered prompts to guide large models through the task step by step. Each round of prompts is dynamically adjusted based on the output of the previous round, forming an interactive reasoning process. This approach has the advantage of reducing the risk of LLM hallucinations (step-by-step verification reduces the probability of the model generating incorrect conclusions at once) and improving interpretability (the output of each round can serve as intermediate evidence to support the final decision).
[0028] Example 1: This paper designs a large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion, which effectively extracts the vulnerability line feature information, semantic information and text information of the code, and can help developers quickly verify the authenticity of the vulnerability and obtain vulnerability solutions. The method includes the following specific steps: 1) Use scripts to collect the vulnerability code field (func_before), repair information field (patch_diff), repaired code field (func_before), vulnerability explanation information field (cve_description), and common defect enumeration type field (cwe_type) in the source code vulnerability detection dataset; 2) Automatically parse and extract vulnerable lines (vulnerable_lines) based on the differences between the code before and after the vulnerability fix (i.e., the func_before and func_after fields); 3) Based on the vulnerability code information obtained in steps 1) and 2), the thought chain reasoning method is used to guide the large model to generate vulnerability explanation information with deeper semantics, enhancing the large model's ability to understand the vulnerability causes and repair methods; 4) The vulnerability code and fix information obtained in step 1), the vulnerability line information obtained in step 2), and the vulnerability explanation information obtained in step 3) are processed separately to construct five enhanced vulnerability code datasets with different feature distributions; 5) Based on the obtained enhanced vulnerability code dataset, the pre-trained Qwen2.5-Coder-7B base model is trained using the parameter efficient fine-tuning technology (LoRA) to generate an independent LoRA adaptation layer to learn different task features such as vulnerability detection and vulnerability explanation; 6) After obtaining the corresponding LoRA adaptation layer for each data set, load the LoRA adaptation layer corresponding to the vulnerability detection to complete the vulnerability detection task; initialize the weight coefficient, use the weight coefficient to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficient using the gradient-free update algorithm to complete the vulnerability explanation task.
[0029] Example 2: This embodiment is further optimized on the basis of the above embodiment. The similarities with the above technical solutions are not repeated here. In order to better implement the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: Step 1) is specifically: for each source code vulnerability detection dataset (such as Big-Vul, Prime-Vul, etc.), a python script is used to automatically extract the vulnerability code field (func_before), repair information field (patch_diff), repaired code field (func_before), vulnerability explanation information field (cve_description) and common defect enumeration type field (cwe_type) in the source code vulnerability detection dataset, and the extracted various information is saved in json format; when extracting, it is noted that the function itself needs to be filtered; the filtering operation includes: deleting incomplete functions, including functions that do not end with} and functions that end with ;; excluding functions that have not been actually modified (no lines are added or deleted); excluding functions that have been modified by more than 70% (this is because the original function and the modified function are no longer strongly related in semantics and syntax due to too many modifications); and excluding functions with less than 5 lines.
[0030] Example 3: This embodiment is further optimized based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting is particularly adopted: Step 2) includes the following specific steps: 2.1) Differential Extraction: Automatically analyze the differences between the vulnerable code and the fixed code using the Python difflib library, recording added and deleted fields and their corresponding line numbers. 2.2) Vulnerability Line Determination: For deleted lines, they are directly marked as vulnerable lines. For lines added in the repaired code, the lines directly associated with the context of adding the line and appearing in the vulnerable code are marked as vulnerable lines. 2.3) Determining Directly Related Lines in Context: Use the source code analysis tool joern to parse the repaired code and generate the corresponding node.csv and egde.csv files for each C / C++ file. In the node.csv file, use regular expression matching to find the target added line. Then, return to egde.csv and use the CONTROL field to determine the lines directly related to the target line. Finally, use the joern tool to parse the vulnerable code and perform regular expression matching on the directly related lines found in the repaired code to locate the corresponding vulnerable line.
[0031] Example 4: This embodiment is further optimized based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting is particularly adopted: Step 3) includes the following specific steps: 3.1) Sequentially extract the vulnerability code field, repair information field, repaired code field, vulnerability explanation information field, general defect enumeration type field, and the code vulnerability line information located in step 2) obtained in step 1), and call the API design prompt word of the large model DeepSeek; The prompt words are as follows: I want you to act as a vulnerability detection expert . This is acodesnippet:{func_before}.lets think step by step. Firstly, what is the purpose of the function in the following code snippet. Please summarize the answer in onesentence. The code snippet above has a vulnerability type : {cwe_type} and thevulnerabilitydescription is :{cve_description}. Then according to thevulnerable description, please only output the vulnerable lines of the codesnippet. The vulnerable lines are {vulnerability_lines} and the correct way to fix it is by {patch_diff}. Now please double-check your answer and only output the repaired code snippet according to the patch diff. The code snippet after refinement is {func_after}.Now pleasereconsider your answer andoutput the vulnerability interpretation based on the above information. DO NOT INCLUDE ANYTHING ELSE EXCEPT VNLNERABILITYINTERPRETATION IN YOUR RESPONSE. 3.2) Enable the DeepSeek historical message mechanism, so that the large model DeepSeek can integrate the prompt word information and its response information from previous rounds to obtain a complete chain of thought, and finally output a more detailed vulnerability explanation information including the cause of the vulnerability triggering and repair information.
[0032] Example 5: This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better realize the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting method is particularly adopted: in the step 4), the five enhanced vulnerability code data sets with different feature distributions constructed are: vulnerability detection (vulnerability detection) data set, vulnerability type detection (vulnerability type) data set, vulnerability line location (vulnerable lines location) data set, vulnerability repair (vulnerability patch) data set, vulnerability interpretation (vulnerability interpretation) data set; wherein, the input (input) of each enhanced vulnerability code data set is the vulnerability code, and the output is the corresponding task target.
[0033] Example 6: This embodiment is further optimized based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, the following setting is particularly adopted: Step 5) is specifically: Use LoRA technology to fine-tune the five enhanced vulnerability code datasets generated in step 4) for the pre-trained Qwen2.5-Coder-7B base model to generate five independent LoRA adaptation layers; during the fine-tuning process, the input data is Instruction and Input, and the output data is the label corresponding to each enhanced vulnerability code dataset task (wherein, the label of the vulnerability detection dataset is 0 / 1 binary classification, representing the presence or absence of a vulnerability; the label of the vulnerability type detection dataset is the general defect enumeration type; the label of the vulnerability line location dataset is the code vulnerability line; the label of the vulnerability repair dataset is the code repair information; the label of the vulnerability explanation dataset is the enhanced vulnerability explanation information obtained in step 3)). Among them, the Input is the vulnerability code to be detected on the five enhanced vulnerability code datasets, and the Instruction is constructed on each enhanced vulnerability code dataset through multiple rounds of prompt chains. The following example: Vulnerability detection: Detect whether the following code is vulnerable or not. Onlyoutput '1'(vulnerable) or '0'(non-vulnerable) Vulnerability type detection: The following code is vulnerable. Please only output itsvulnerability type with the format: CWE-416 Vulnerability explanation: The following code is vulnerable and its vulnerability typeis CWE-476. Please output its vulnerability interpretation in one sentence. Vulnerable line location: The following vulnerable code with a vulnerable type CWE-77 has a vulnerability interpretation: picocombefore 2.0 has a command injection vulnerability in the'send and receive file' command because the command line is executed by / bin / sh unsafely.. Please output its vulnerable lines with the following format: {'line_number' : 'Vulnerable line'} Vulnerability fix: The following vulnerable code with a vulnerable type has a vulnerability interpretation: Multiple unspecified vulnerabilities in Google Chrome before 36.0.1985.143 allow attackers to cause a denial of service or possibly have other impact via unknown vectors.. Its vulnerable lines are {'8':'return narrowPrecisionToFloat(sqrt((pow(ctm.xScale(), 2)+ pow(ctm.yScale(), 2)) / 2));'}. Please only output the lines needed to be added or deleted to make the code correct with the following format: {'add':[lines to be added], 'deleted':[lines to be deleted]}"; Hyperparameter selection: During fine-tuning, considering performance factors, this invention also uniformly padded / truncated all input data to a length of 4096 tokens. Different LoRA rank values (LoRA rank (r)) were set for different tasks: the LoRA rank for the vulnerability detection task was set to 8, and the LoRA rank for other generation tasks was set to 16. In other words, the LoRA rank (r) set in this invention is different: for the vulnerability detection task, r = 8, and for the other generation tasks, r = 16.
[0034] Example 7: This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better realize the large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion described in the present invention, in order to achieve multi-task fusion capability, after training for different tasks respectively, in the vulnerability interpretation task, the LoRA adaptation layers of multiple tasks are integrated into a unified output path in a weighted manner by constructing a dynamic fusion controller, thereby improving the interpretation effect. In particular, the following setting method is adopted: in the step 6), when completing the vulnerability detection task, the vulnerability detection LoRA adaptation layer is loaded using the transformers library of Python to complete the vulnerability detection task; Initialize the weight coefficients, use the weight coefficients to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficients using a gradient-free update algorithm to complete the vulnerability explanation task. The specific steps include the following: 6.2.1) Weight coefficient initialization: Use the cosine similarity of the task to initialize the weight of each LoRA adaptation layer, and then build a parameterized dynamic fusion controller. The specific formula is as follows: ; in, That is, the learnable fusion parameters can be dynamically adjusted through mechanisms or algorithms; That is, the original parameters of the model; Parameters of each LoRA module; 6.2.2) Gradient-Free Optimization Algorithm: The Cross Entropy Method (CEM) algorithm is used for coefficient optimization. CEM is a stochastic optimization algorithm that approaches the optimal solution by sampling from a probability distribution, evaluating the sample quality, and updating the distribution parameters. The evaluation criterion (loss function) for vulnerability explanation quality is defined as follows: ; Among them, IOU (Intersection over Union, positioning accuracy) is used to measure the overlap between the predicted area and the real area, and BLEU-4 (Bilingual Evaluation Understudy) is used to measure the matching degree between the text generated by the model and the reference text (ground truth). In this invention, A and B are set to 0.5 and 0.5 respectively; 6.2.3) Fusion coefficient update: Using the CEM algorithm constructed in 6.2.2), sample and evaluate a small number of vulnerability explanation samples (1% in this paper) to update the fusion parameters. , realizing the dynamic fusion of different LoRA modules and finally outputting the vulnerability explanation; achieving the goal of enabling the pre-trained Qwen2.5-Coder-7B base model to load the LoRA adaptation layer parameters fine-tuned by different data sets, so that the pre-trained Qwen2.5-Coder-7B base model can obtain knowledge from different fields, effectively improving the performance and adaptability of the model in the field of vulnerability explanation.
[0035] Example 8: This embodiment is further optimized based on any of the above embodiments, and the similarities with the above technical solutions are not repeated here. Figure 1 As shown in the figure, a large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion can fully capture the semantic, grammatical and structural features of code data while performing data enhancement, providing richer training data for model learning. Figure 1 As shown, the following steps are included: The first step is data preprocessing, which involves performing a preliminary analysis of the vulnerability detection dataset. For each source code vulnerability detection dataset (such as Big-Vul, Prime-Vul, etc.), a script is used to collect the vulnerability code field (func_before), the repair information field (patch_diff), the repaired code field (func_before), the vulnerability explanation information field (cve_description), and the common defect enumeration type field (cwe_type) in the source code vulnerability detection dataset. The extracted information is then saved in JSON format. During the extraction, please note that the functions themselves must be filtered. The filtering operations include: deleting incomplete functions, including functions that do not end with} and functions that end with ;; excluding functions that have not been actually modified (no lines have been added or deleted); excluding functions that have been modified by more than 70% (this is because the original and modified functions are no longer strongly related in semantics or syntax due to excessive modifications); and excluding functions with fewer than 5 lines.
[0036] The second step is to locate the vulnerable lines. First, perform differential extraction: use Python's difflib library to automatically parse the differences between the vulnerable code and the repaired code, record the added and deleted fields and the corresponding line number information; then preliminarily determine the vulnerable lines: for deleted lines, directly mark them as vulnerable lines; for lines added in the repaired code, mark the context-directed lines of the added lines and the lines that appear in the vulnerable code as vulnerable lines; finally, locate the context-directed lines: use the source code detection tool joern to parse the repaired code, and obtain the corresponding node.csv and egde.csv files for each c / c++ file; find the target added lines in the node.csv file based on regular matching, and then go back to egde.csv to determine the lines directly related to the target lines based on the CONTROL field; finally, use the joern tool to parse the vulnerable code, perform regular matching on the directly related lines found in the repaired code, and determine the corresponding vulnerable lines.
[0037] Step 3: Generate detailed vulnerability explanation information. Sequentially extract the vulnerability code field (func_before), the patch information field (patch_diff), the patched code field (func_before), the vulnerability explanation field (cve_description), the common defect enumeration type field (cwe_type), and the code vulnerability line information located in step 2). Using the concept of a multi-round prompt chain, call the official API of the large model DeepSeek to design the following prompt words: I want you to act as a vulnerability detection expert . This is acodesnippet:{func_before}.lets think step by step. Firstly, what is the purpose of the function in the following code snippet. Please summarize the answer in onesentence. The code snippet above has a vulnerability type : {cwe_id} and thevulnerabilitydescription is :{cve_description}. Then according to thevulnerable description, please only output the vulnerable lines of the codesnippet. The vulnerable lines are {vulnerability_lines} and the correct way to fix it is by {patch_diff}. Now please double-check your answer and only output the repaired code snippet according to the patch diff. The code snippet after refinement is {func_after}.Now pleasereconsider your answer andoutput the vulnerability interpretation based on the above information. DO NOT INCLUDE ANYTHING ELSE EXCEPT VNLNERABILITYINTERPRETATION IN YOUR RESPONSE. Then, according to Deepseek's official documentation, we enabled the Deepseek historical message mechanism. This enabled the large-scale Deepseek model to integrate the prompt word information and its response information from previous rounds to obtain a complete chain of thought. Finally, it output a more detailed vulnerability explanation information including the cause of the vulnerability triggering and repair information.
[0038] Step 4: Data Augmentation: After obtaining the vulnerability code information from steps 1, 2, and 3, we constructed five enhanced vulnerability code datasets with different feature distributions: vulnerability detection dataset, vulnerability type detection dataset, vulnerability line location dataset, vulnerability patch dataset, and vulnerability interpretation dataset. The input for each enhanced vulnerability code dataset is the vulnerability code, and the output is the label for the corresponding dataset task.
[0039] Step 5: Fine-tuning the Large Model LoRA: Using LoRA technology, fine-tune the five enhanced vulnerability code datasets generated in Step 4 against the pre-trained Qwen2.5-Coder-7B base model to generate five independent LoRA adaptation layers. During the fine-tuning process, the input data is the prompt (Instruction) and the input (Input). The Input is the vulnerability code to be detected in each of the five enhanced vulnerability code datasets. The Instruction is different for different enhanced vulnerability code datasets. The present invention still uses multiple rounds of prompt chains to generate Instructions: Vulnerability detection: Detect whether the following code is vulnerable or not. Onlyoutput '1'(vulnerable) or '0'(non-vulnerable) Vulnerability type detection: The following code is vulnerable. Please only output itsvulnerability type with the format: CWE-416 Vulnerability explanation: The following code is vulnerable and its vulnerability typeis CWE-476. Please output its vulnerability interpretation in one sentence. Vulnerable line location: The following vulnerable code with a vulnerable type CWE-77 has a vulnerability interpretation: Picocombefore 2.0 has a command injection vulnerability in the'send and receive file' command because the command line is executed by / bin / sh unsafely.. Please output its vulnerable lines with the following format: {'line_number' : 'Vulnerable line'} Vulnerability fix: The following vulnerable code with a vulnerable type has a vulnerability interpretation: Multiple unspecified vulnerabilities in Google Chrome before 36.0.1985.143 allow attackers to cause a denial of service or possibly have other impact via unknown vectors.. Its vulnerable lines are {'8':'return narrowPrecisionToFloat(sqrt((pow(ctm.xScale(), 2)+ pow(ctm.yScale(), 2)) / 2));'}. Please only output the lines needed to be added or deleted to make the code correct with the following format: {'add':[lines to be added], 'deleted':[lines to be deleted]}", The output data is the labels corresponding to each enhanced vulnerability code dataset task (the label of the vulnerability detection dataset is a binary classification of 0 / 1, representing the presence or absence of a vulnerability; the label of the vulnerability type detection dataset is a general defect enumeration type; the label of the vulnerability line location dataset is the code vulnerability line; the label of the vulnerability repair dataset is the code repair information; and the label of the vulnerability explanation dataset is the enhanced vulnerability explanation information obtained in step 3).
[0040] Hyperparameter selection: During fine-tuning, considering performance factors, the present invention also uniformly padded / truncated all input data to a length of 4096 tokens. For different tasks, the present invention sets the rank (r) of LoRA differently: for vulnerability detection tasks, r=8, and for other generation tasks, r=16.
[0041] Step 6: Use the LoRA fusion mechanism to complete vulnerability detection and explanation tasks.
[0042] Vulnerability detection: Use Python's transformers library to load the vulnerability detection LoRA adaptation layer to complete the vulnerability detection task.
[0043] Vulnerability Explanation Task: To integrate the knowledge capabilities of multiple tasks, a dynamic fusion controller is introduced in this task. The LoRA adaptation layers trained from multiple tasks are combined into a module in the form of learnable weights. The specific steps are as follows: 1) Weight coefficient initialization: Use the cosine similarity of the task to initialize the weight of each LoRA adaptation layer, and then build a parameterized dynamic fusion controller. The specific formula is as follows: ; in, That is, the learnable fusion parameters can be dynamically adjusted through mechanisms or algorithms; That is, the original parameters of the model; Parameters of each LoRA module; 2) Gradient-free update algorithm implementation: The CEM (Cross Entropy Method) algorithm is used for coefficient optimization. The evaluation criteria (loss function) for vulnerability explanation quality defined in this invention are as follows: ; The Intersection over Union (IOU) is used to measure the overlap between the predicted and true regions, and the Bilingual Evaluation Understudy (BLEU-4) is used to measure the match between the model-generated text and the reference text (ground truth). In this paper, A and B are set to 0.5 and 0.5, respectively.
[0044] 3) Fusion coefficient update: Using the CME algorithm in step 2), a small amount of vulnerability explanation data set (1% in this paper) is sampled and updated. , enabling the pre-trained Qwen2.5-Coder-7B base model to load the LoRA adaptation layer parameters fine-tuned by different data sets, so that the pre-trained Qwen2.5-Coder-7B base model can obtain knowledge from different fields, effectively improving the performance and adaptability of the model in the field of vulnerability interpretation.
[0045] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.
Claims
1. A large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion, characterized by: The specific steps include: 1) Use scripts to collect vulnerability code fields, repair information fields, repaired code fields, vulnerability explanation information fields, and general defect enumeration type fields in the source code vulnerability detection dataset; 2) Automatically parse and extract code vulnerability line information based on the differences before and after vulnerability repair; 3) Based on the vulnerability code information obtained in steps 1) and 2), the thought chain reasoning method is used to guide the large model to generate vulnerability explanation information with deeper semantic depth; 4) The vulnerability code and fix information obtained in step 1), the vulnerability line information obtained in step 2), and the vulnerability explanation information obtained in step 3) are processed separately to construct five enhanced vulnerability code datasets with different feature distributions; 5) Based on the obtained enhanced vulnerability code dataset, the pre-trained Qwen2.5-Coder-7B base model is trained using efficient parameter fine-tuning technology to generate an independent LoRA adaptation layer; 6) After obtaining the corresponding LoRA adaptation layer for each enhanced vulnerability code dataset, load the LoRA adaptation layer corresponding to the vulnerability detection to complete the vulnerability detection task; initialize the weight coefficient, use the weight coefficient to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficient using the gradient-free update algorithm to complete the vulnerability explanation task.
2. A large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1, characterized in that: Step 1) specifically includes: for each source code vulnerability detection dataset, using a Python script to automatically extract the vulnerability code field, repair information field, repaired code field, vulnerability explanation information field, and general defect enumeration type field in the source code vulnerability detection dataset, and saving the extracted information in JSON format; During extraction, the function itself is filtered. The filtering operation includes: deleting incomplete functions, including functions that do not end with} and functions that end with ;; excluding functions that have not been actually modified; excluding functions that have been modified by more than 70%; and excluding functions with less than 5 lines.
3. The large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1 is characterized in that: The step 2) includes the following specific steps: 2.1) Differential Extraction: Automatically analyze the differences between the vulnerable code and the fixed code using the Python difflib library, recording added and deleted fields and their corresponding line numbers. 2.2) Vulnerability Line Determination: For deleted lines, they are directly marked as vulnerable lines. For lines added in the repaired code, the lines directly associated with the context of adding the line and appearing in the vulnerable code are marked as vulnerable lines. 2.3) Determining the context-sensitive lines: Use the source code analysis tool joern to parse the repaired code and generate the corresponding node.csv and egde.csv files for each C / C++ file. In the node.csv file, find the target row based on regular expression matching. Then, return to egde.csv and use the CONTROL field to determine the rows directly related to the target row. Finally, use the joern tool to parse the vulnerable code and perform regular expression matching on the directly related rows found in the repaired code to find the corresponding vulnerable row.
4. A large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1, 2 or 3, characterized in that: The step 3) includes the following specific steps: 3.1) Sequentially extract the vulnerability code field, repair information field, repaired code field, vulnerability explanation information field, general defect enumeration type field, and the code vulnerability line information located in step 2) obtained in step 1), and call the API design prompt word of the large model DeepSeek; 3.2) Enable the DeepSeek historical message mechanism, so that the domestic large-scale DeepSeek model can integrate the information of previous rounds of prompt words to obtain a complete chain of thought, and finally output more detailed vulnerability explanation information including the cause of the vulnerability triggering and repair information.
5. The large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1 is characterized in that: In step 4), the five enhanced vulnerability code datasets with different feature distributions are constructed: vulnerability detection dataset, vulnerability type detection dataset, vulnerability line location dataset, vulnerability repair dataset, and vulnerability explanation dataset; wherein the input of each enhanced vulnerability code dataset is the vulnerability code, and the output is the corresponding task objective.
6. The large model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1 is characterized in that: The step 5) is specifically as follows: using LoRA technology to fine-tune the five enhanced vulnerability code data sets generated in step 4) for the pre-trained Qwen2.5-Coder-7B base model to generate five independent LoRA adaptation layers; during the fine-tuning process, the input data is Instruction and Input, and the output data is the label corresponding to each enhanced vulnerability code data set task; wherein, Input is the vulnerability code to be detected on the five enhanced vulnerability code data sets, and Instruction is constructed separately on each enhanced vulnerability code data set through multiple rounds of prompt chains; during the fine-tuning process, all input data are uniformly padded or truncated to a length of 4096 tokens; different LoRA rank values are set for different tasks: wherein, the LoRA rank of the vulnerability detection task is set to 8, and the LoRA rank of other generation tasks is set to 16.
7. The large-model vulnerability detection and interpretation method based on data enhancement and LoRA fusion according to claim 1 is characterized in that: In step 6), when the vulnerability detection task is completed, the vulnerability detection LoRA adaptation layer is loaded using the transformers library of Python to complete the vulnerability detection task; Initialize the weight coefficients, use the weight coefficients to merge multiple LoRA adaptation layers, and use a small number of samples to update the weight coefficients using a gradient-free update algorithm to complete the vulnerability explanation task. The specific steps include the following: 6.2.1) Weight coefficient initialization: Use the cosine similarity of the task to initialize the weight of each LoRA adaptation layer, and then build a parameterized dynamic fusion controller. The specific formula is as follows: ； in, That is, the learnable fusion parameters can be dynamically adjusted through mechanisms or algorithms; That is, the original parameters of the model; Parameters of each LoRA module; 6.2.2) Gradient-Free Optimization Algorithm: The CEM algorithm is used for coefficient optimization. The evaluation criteria for vulnerability explanation quality are defined as follows: ； Among them, IOU is used to measure the overlap between the predicted area and the true area, and BLEU-4 is used to measure the matching degree between the text generated by the model and the reference text. A and B are taken as 0.5 and 0.5 respectively; 6.2.3) Fusion coefficient update: Use the CEM algorithm constructed in step 6.2.2) to sample and evaluate a small number of vulnerability explanation samples and update the fusion parameters , realize the dynamic fusion of different LoRA modules, and finally output the vulnerability explanation.