A method and system for intelligent mining of software security vulnerabilities based on large language model

Abstract

The present invention belongs to the field of software security technology, and its purpose is to provide a method and system for intelligent mining of software security vulnerabilities based on a large language model. The method in the present invention includes: obtaining a preprocessed vulnerability code data set; obtaining CWE vulnerability data, and constructing a CWE vulnerability library based on the CWE vulnerability data; constructing a program dependency graph based on the preprocessed vulnerability code data set; creating a sample data set based on the program dependency graph, obtaining a pre-trained large language model, and using the sample data set, using a lightweight fine-tuning method to train the pre-trained large language model to obtain a software security vulnerability intelligent mining model; receiving a code to be tested, and performing vulnerability mining processing on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain a vulnerability mining result. The present invention can automatically, efficiently and accurately mine potential security vulnerabilities in software source code, and can obtain complete vulnerability mining results.

Description

Technical Field

[0001] The present invention belongs to the field of software security technology, and in particular relates to a method and system for intelligently mining software security vulnerabilities based on a large language model. Background Art

[0002] With the rapid development of software technology, software vulnerabilities have become a potential threat to network security. These vulnerabilities may exist due to flaws in the development process or design defects, providing potential intrusion pathways for criminals. Software vulnerability mining technology can timely discover security vulnerabilities in the system by tracking the data generated during the software operation and analyzing the code executed by the monitoring program, preventing hackers from exploiting these vulnerabilities to attack, and ensuring the user's information security and the stable operation of the system. Therefore, the importance of software vulnerability mining lies in timely discovering and repairing potential threats and providing users with effective security protection.

[0003] In the prior art, for example, the invention application with application number CN202111612183.9 discloses "Vulnerability Mining System and Method Based on Intermediate Language and Transfer Representation Learning", which proposes a code vulnerability mining system based on BiLSTM (bidirectional long short-term memory) neural network and convolutional neural network. The system converts high-level programming languages ​​into intermediate languages ​​for representation, uses BiLSTM networks to extract code features, and then uses convolutional neural networks to determine whether the code has vulnerabilities.

[0004] However, in the process of using the prior art, the inventors found that the prior art has at least the following problems:

[0005] Traditional code vulnerability mining methods based on deep learning have poor interpretability and cannot understand the semantics and context of the code. They can only determine whether there is a vulnerability but cannot provide accurate vulnerability classification information. Summary of the invention

[0006] The present invention aims to solve the above technical problems at least to a certain extent. The present invention provides a method, system, device and product for intelligent mining of software security vulnerabilities based on a large language model.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for intelligently mining software security vulnerabilities based on a large language model, comprising:

[0009] Acquire a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set;

[0010] Acquire CWE vulnerability data, and construct a CWE vulnerability library based on the CWE vulnerability data;

[0011] Constructing a program dependency graph based on the preprocessed vulnerability code dataset;

[0012] A sample data set is created according to the program dependency atlas, a pre-trained large language model is obtained, and the pre-trained large language model is trained using the sample data set and a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model;

[0013] The code to be tested is received, and vulnerability mining processing is performed on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain a vulnerability mining result.

[0014] In a possible design, the vulnerability code data set includes a plurality of code middlewares; correspondingly, the vulnerability code data set is preprocessed to obtain a preprocessed vulnerability code data set, including:

[0015] Based on the MinHash algorithm, obtain the code snippet of each code middleware in the vulnerability code data set, and construct a code snippet set based on the code snippet of each code middleware, and obtain the MinHash signature of each code snippet in the code snippet set;

[0016] Based on the LSH algorithm, the MinHash signature of each code snippet in the code snippet set is used as a high-dimensional vector to construct an LSH hash table;

[0017] The MinHash signatures in the same hash table position in the LSH hash table are calculated similarity pair by pair. When the calculated similarity is higher than a preset threshold, a representative signature is retained and the remaining similar MinHash signatures are removed to obtain the deduplicated MinHash signatures. The deduplicated MinHash signatures are then mapped back to the corresponding code snippets to construct a preprocessed vulnerability code dataset.

[0018] In a possible design, the preprocessed vulnerability code dataset includes a plurality of deduplicated code middlewares; correspondingly, a program dependency graph is constructed based on the preprocessed vulnerability code dataset, including:

[0019] Each deduplicated code middleware in the preprocessed vulnerability code data set is constructed as an abstract syntax tree, and a code basic block is extracted from each abstract syntax tree, and a control flow relationship between code basic blocks is extracted according to the control flow transfer statements in each abstract syntax tree;

[0020] Taking the code basic blocks as nodes and the control flow relationships as control flow edges, a control flow graph corresponding to each deduplicated code middleware is constructed;

[0021] The data flow dependency relationship of each node in the control flow graph is obtained, and the data flow dependency relationship is used as the data dependency edge. On the basis of the control flow graph, a program dependency graph corresponding to each deduplicated code middleware is constructed, and each program dependency graph constitutes a program dependency graph set.

[0022] In a possible design, after constructing the program dependency graph, the method further includes:

[0023] The program dependency graph is pruned to obtain pruned program dependency graphs corresponding to each deduplicated code middleware, so as to construct a sample data set based on each pruned program dependency graph.

[0024] In a possible design, a sample data set is created based on the program dependency atlas, including:

[0025] Assign vulnerability labels to the program dependency graphs in the program dependency graph set respectively, and bind each vulnerability label to its corresponding program dependency graph;

[0026] A sample data set is created based on each program dependency graph bound with a vulnerability label; wherein the sample data set includes a training set, a test set and a validation set obtained by dividing each program dependency graph bound with a vulnerability label according to a preset ratio.

[0027] In a possible design, the pre-trained large language model adopts the DeepSeek-Coder-6.7B-instruct large language model, and a classification head is arranged on the top of the DeepSeek-Coder-6.7B-instruct large language model; the lightweight fine-tuning method adopts the P-Tuning fine-tuning method.

[0028] In a possible design, the vulnerability mining result includes a vulnerability label prediction result and specified vulnerability information; correspondingly, vulnerability mining processing is performed on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain a vulnerability mining result, including:

[0029] Obtaining a dependency graph of a program under test corresponding to the code under test;

[0030] Inputting the dependency graph of the program to be tested into the software security vulnerability intelligent mining model, so that the software security vulnerability intelligent mining model performs inference prediction on the dependency graph of the program to be tested to obtain vulnerability label prediction results;

[0031] Query the CWE vulnerability database for specified vulnerability information that matches the vulnerability label prediction result.

[0032] In a second aspect, the present invention provides a software security vulnerability intelligent mining system based on a large language model, which is used to implement the software security vulnerability intelligent mining method based on a large language model as described in any one of the above; the software security vulnerability intelligent mining system based on a large language model includes:

[0033] A sample data set acquisition module is used to acquire a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set; it is also used to construct a program dependency graph based on the preprocessed vulnerability code data set, and create a sample data set based on the program dependency graph;

[0034] A model training module is connected to the sample data set acquisition module for acquiring a pre-trained large language model, and uses the sample data set to train the pre-trained large language model using a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model;

[0035] A CWE vulnerability library construction module is used to obtain CWE vulnerability data and construct a CWE vulnerability library based on the CWE vulnerability data;

[0036] The vulnerability mining module is communicatively connected with the model training module and the CWE vulnerability library construction module, and is used to receive the code to be tested, and perform vulnerability mining processing on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results.

[0037] In a third aspect, the present invention provides an electronic device, comprising:

[0038] a memory for storing computer program instructions; and,

[0039] A processor is used to execute the computer program instructions to complete the operation of the software security vulnerability intelligent mining method based on a large language model as described in any one of the above.

[0040] In a fourth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for intelligent mining of software security vulnerabilities based on a large language model as described in any one of the above.

[0041] The beneficial effects of the present invention are:

[0042] The present invention discloses a method, system, device and product for intelligent mining of software security vulnerabilities based on a large language model, which can automatically, efficiently and accurately mine potential security vulnerabilities in software source code, and can obtain complete vulnerability mining results. Specifically, the present invention adopts a pre-trained large language model as the initial model of the intelligent mining model of software security vulnerabilities, abstractly represents the code data in the vulnerability code data set as a program dependency graph, and creates a sample data set based on the program dependency graph set, and then uses the sample data set to perform lightweight fine-tuning training on the pre-trained large language model, so as to realize intelligent mining of software security vulnerabilities of the code to be tested based on the obtained intelligent mining model of software security vulnerabilities. In this process, the present invention uses the large language model's powerful understanding ability of natural language and code language to realize intelligent mining of software security vulnerabilities, and can better capture the dependency relationship, call relationship and logical flow in the code to be tested, which helps to identify potential vulnerabilities hidden behind the code, thereby improving the mining ability of the present invention for potential security vulnerabilities in software code. In addition, traditional code vulnerability detection methods can only detect whether the code has vulnerabilities, but cannot classify the vulnerabilities and give vulnerability descriptions because they only perform binary classification training on the model or are limited by their neural network performance. However, the present invention collects open source vulnerability data to construct a CWE vulnerability library before performing vulnerability mining on the code to be tested. Therefore, when the software security vulnerability intelligent mining model is used to perform vulnerability mining on the code to be tested, the prediction results of the large language model are associated with the data in the CWE vulnerability library. When potential vulnerabilities are detected in the code to be tested, the vulnerabilities can be further identified based on the CWE vulnerability library to obtain vulnerability mining results including detailed descriptions of the vulnerabilities, vulnerability impacts, and mitigation measures. This can better discover security issues in software codes so that users and software manufacturers can repair them in a timely manner.

[0043] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of the software security vulnerability intelligent mining method based on a large language model in Example 1;

[0045] Figure 2 is a schematic diagram of preprocessing the vulnerability code data set in Example 1;

[0046] Figure 3 It is a schematic diagram of constructing a program dependency graph from the vulnerability code data set after preprocessing in Example 1;

[0047] Figure 4 is a schematic diagram of training the pre-trained large language model in Example 1;

[0048] Figure 5 It is a module block diagram of the software security vulnerability intelligent mining system based on the large language model in Example 2;

[0049] Figure 6 This is a module block diagram of the electronic device in Example 3. DETAILED DESCRIPTION

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0051] Embodiment 1:

[0052] This embodiment discloses a method for intelligent mining of software security vulnerabilities based on a large language model, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.

[0053] like Figure 1 As shown, a method for intelligently mining software security vulnerabilities based on a large language model may include, but is not limited to, the following steps:

[0054] S1. Obtain a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set; it should be noted that, in this embodiment, the vulnerability code data set includes multiple source code files, and each source code file can be converted into a code middleware, and each code middleware includes a group of semantically related code fragments. It should be understood that the code middleware generated by the same source code file has a data flow dependency or a control dependency. Based on this, the source code files in the vulnerability code data set can be standardized by applying regular expressions (Regular Expression, which is an expression used to describe and match strings, and is a powerful text processing tool that can be used to format source code) including deleting comments and non-ASCII characters (a computer encoding system based on Latin letters, mainly used to display modern English and other Western European languages), extracting all function and variable definitions and their usage in the source code file, and each code middleware in the source code file is bound to an initial label of whether there is a vulnerability. If the label value of the initial label is 0, it indicates that the code middleware has no vulnerability, and if the label value is 1, it indicates that the code middleware has a vulnerability.

[0055] Specifically, in this embodiment, the vulnerability code data set adopts the SARD (Software Assurance Reference Dataset) data set of the National Public Vulnerability Database of the United States published by NIST (National Institute of Standards and Technology). It should be noted that the SARD data set contains a large number of known vulnerability samples in real software systems, and these vulnerability samples come from multiple sources, including publicly reported vulnerabilities, known security incidents, and software versions with fixed vulnerabilities. It covers various programming languages ​​and application types, including five programming languages ​​​​C, C++, Java, PHP and C#, covering more than 150 different types of vulnerabilities, including but not limited to: buffer overflow type vulnerability (CWE-121), out-of-bounds write type vulnerability (CWE-787), improper input validation type vulnerability (CWE-20), operating system command injection type vulnerability (CWE-78), server request forgery type vulnerability (CWE-918) and path traversal type vulnerability (CWE-22), etc. Since the training data used in traditional code vulnerability mining solutions often only covers a certain programming language and can only detect vulnerabilities in a specific programming language, the generalization ability for other programming languages ​​is poor. This embodiment improves the diversity of sample data sets by using the SARD data set as a vulnerability code data set, and can make the model obtained after training sensitive to multiple high-level programming languages, thereby improving the generalization ability for different programming languages ​​during vulnerability mining.

[0056] In this embodiment, the vulnerability code data set includes a plurality of code middlewares; correspondingly, in step S1, the vulnerability code data set is preprocessed to obtain a preprocessed vulnerability code data set, such as Figure 2 As shown, including:

[0057] S101. Based on the MinHash (minimum hash) algorithm, obtain the code snippet of each code middleware in the vulnerability code data set, construct a code snippet set based on the code snippet of each code middleware, and obtain the MinHash signature of each code snippet in the code snippet set; wherein the code snippet set can be expressed as: , are all the code snippets in each code middleware, and the first The code middleware can be expressed as Specifically, in this embodiment, for , you can use A hash function is used to map it to a hash value of fixed length, and the minimum value is selected from all the generated hash values ​​as the code snippet. MinHash value .

[0058] It should be noted that the MinHash algorithm is a hashing technology that can be used to effectively estimate the similarity between sets. In the implementation process, the similarity between two sets can be quickly compared by hashing the elements in the set into a smaller signature (usually a fixed-length integer or bit string, which is the MinHash signature in this article).

[0059] S102. Based on the LSH (Locality-Sensitive Hashing) algorithm, the MinHash signature of each code snippet in the code snippet set is used as a high-dimensional vector to construct an LSH hash table; specifically, the LSH hash table includes a hash function family Based on the hash functions in the hash function family, the MinHash signature can be mapped to one or more hash table locations (also called "buckets"), and each hash table location includes multiple MinHash signatures that match the corresponding hash function. Specifically, the construction process of the LSH hash table can be expressed as:

[0060]

[0061] in, is the ith code snippet, is a collection of all code snippets, It is Hash functions The hash table location mapped to, is an identifier for a hash table location.

[0062] It should be noted that the LSH algorithm refers to an approximate search technology used to quickly search for similar items in a high-dimensional space, and is suitable for processing large-scale data sets.

[0063] S103. Perform similarity calculation on the MinHash signatures in the same hash table position in the LSH hash table one by one. When the calculated similarity is higher than a preset threshold, retain a representative signature and remove the remaining similar MinHash signatures to obtain deduplicated MinHash signatures. Then map the deduplicated MinHash signatures back to the corresponding code snippets to construct a preprocessed vulnerability code dataset.

[0064] In addition, step S1 of this embodiment may further include:

[0065] S104. Using symbolic equivalents to replace the names defined by the user for the vulnerability code dataset, so as to standardize the defined names.

[0066] Specifically, in step S104, Substitute the user-defined function name (or if there are multiple functions, you can use consecutive natural numbers as Suffixes such as and ), you can also use Substitute a user-assigned variable name (or, if there are multiple variables, use consecutive natural numbers as Suffixes such as and ), thereby achieving standardized processing of user definitions.

[0067] Since duplicate data has a great impact on the subsequent fine-tuning of the pre-trained large language model, if the final model is to have better software security vulnerability intelligent mining performance, the vulnerability code data set must be deduplicated before being used to train the pre-trained large language model. However, performing data deduplication on large-scale data is a very time-consuming task. The all-to-all comparison method in the traditional method (i.e., comparing whether any two sets are similar) is very time-consuming and has high time complexity. Specifically, as the number of sets increases, the required calculation time increases quadratically. In this embodiment, according to the above steps S101 to S103, based on the MinHash algorithm and the LSH algorithm, the code middleware in the vulnerability code data set is quickly deduplicated to clean up the duplicate code middleware, and the user-defined names corresponding to the vulnerability code data set can also be standardized based on step S104 to obtain the pre-processed vulnerability code data set. Based on the setting of the above-mentioned preprocessing scheme in this embodiment, due to the use of the LSH algorithm, codes with high similarity are hashed into the same bucket, and only the codes hashed into the same bucket are compared for similarity using the MinHash algorithm, which greatly reduces the number of comparisons and thus reduces the overall time complexity of the deduplication processing.

[0068] S2. Acquire CWE (Common Weakness Enumeration) vulnerability data, and construct a CWE vulnerability library based on the CWE vulnerability data; it should be noted that in this embodiment, the CWE vulnerability data can be stored in a MySQL database, and a CWE vulnerability library can be constructed based on the CWE vulnerability data in the MySQL database to implement the call of CWE vulnerability data. It should also be noted that each CWE vulnerability data contains information such as the CWE number and its description, integrity impact, confidentiality impact, and availability impact. In this embodiment, the CWE official website is used as the data source of the CWE vulnerability library, that is, the CWE vulnerability data is obtained based on the CWE official website.

[0069] S3. Construct a program dependency graph based on the preprocessed vulnerability code dataset; this step is also used to implement code abstract representation.

[0070] In this embodiment, the preprocessed vulnerability code data set includes multiple deduplicated code middlewares; correspondingly, in step S3, a program dependency graph is constructed based on the preprocessed vulnerability code data set, such as Figure 3 As shown, including:

[0071] S301. construct each deduplicated code middleware in the preprocessed vulnerability code data set into an abstract syntax tree (Abstract Syntax Tree, AST), extract code basic blocks (BasicBlocks) from each abstract syntax tree, and extract the control flow relationship between the code basic blocks according to the control flow transfer statements (such as conditional statements, loop statements, etc.) in each abstract syntax tree;

[0072] It should be noted that the abstract syntax tree is a tree representation of the ordered grammatical structure of the source code. It can represent the grammatical structure of the programming language in the form of a tree, and each node in the tree represents a structure in the source code. In addition, the abstract syntax tree only represents the grammatical structure of the code and does not display the semantic association of the code.

[0073] Specifically, in this embodiment, for the deduplicated code middleware of Python language, the ast module is used to generate its abstract syntax tree; for the deduplicated code middleware of JavaScript language, Esprima is used to generate the abstract syntax tree of JavaScript code; for the deduplicated code middleware of C language, libclang or gcc is used to generate the abstract syntax tree.

[0074] S302. Using the code basic blocks as nodes and the control flow relationships as control flow edges, a control flow graph (CFG) corresponding to each deduplicated code middleware is constructed;

[0075] It should be noted that the control flow graph represents the graphical structure of the control flow in the program, in which the nodes represent the basic blocks of code, and the control flow edges represent the control flow relationship between the basic blocks of code. The control flow graph can infer and traverse the possible paths during the execution of the program. Specifically, the control flow graph will select and display different paths after the conditional statement, and represent various control flow relationships by generating directed edges with control significance, thereby associating the code semantic information.

[0076] S303. Obtain the data flow dependency of each node in the control flow graph, and use the data flow dependency as the data dependency edge, and construct a program dependency graph (PDG) corresponding to each deduplicated code middleware on the basis of the control flow graph, and each program dependency graph constitutes a program dependency graph set. It should be understood that the number of program dependency graphs is the same as the number of deduplicated code middleware in the preprocessed vulnerability code data set. Specifically, in this step, the code basic blocks in the control flow graph can be traversed first, and the control flow relationship between each code basic block can be analyzed, such as the definition, use and modification of variables. For example, when a variable is defined or modified in one code basic block and used or modified in another code basic block, a data dependency edge is established to connect the two code basic blocks. Afterwards, when establishing the data flow dependency, it is necessary to consider the arrival definition of the variable in each code basic block. This step involves analyzing the data flow propagation path in the control flow graph to determine the definition point of the variable in each code basic block. Finally, the code basic blocks and the control flow relationships and data flow dependencies between them are represented as a program dependency graph, and the nodes in the program dependency graph represent the code basic blocks, and the edges represent the control flow relationships and data flow dependencies.

[0077] It should be noted that the program dependency graph is used to represent the relationship between basic blocks of code. Unlike the control flow graph that only represents the control flow dependency, the program dependency graph can further represent the data flow dependency of the code through directed edges. When a variable, function or object is defined or modified at the source node and used or modified at the target node, the two statement nodes can be connected through a data dependency edge.

[0078] In step S3, after the program dependency graph is constructed based on step S303, the method further includes:

[0079] S304. Prune the program dependency graph to obtain pruned program dependency graphs corresponding to each deduplicated code middleware, so as to construct a sample data set based on each pruned program dependency graph. It should be noted that, in this embodiment, pruning the program dependency graph can retain important information such as data flow dependency and control flow relationship of the program dependency graph, while reducing the size of the entire graph and the scope of program analysis, which is conducive to reducing the training overhead of the pre-trained large language model in subsequent steps.

[0080] In this embodiment, based on the above steps S101-S104 and steps S301-S304, the source code is converted into mutually dependent code fragments, and then converted into an abstract syntax tree, a program control flow graph, and a program dependency graph in sequence, and finally pruned. This source code processing method retains the control flow and data flow relationships between program source codes, increases the consideration of code context in vulnerability mining, and helps to improve the ability to subsequently mine software code security vulnerabilities.

[0081] S4. Create a sample data set based on the program dependency graph, obtain a pre-trained large language model, and use the sample data set to train the pre-trained large language model using a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model; it should be noted that the large language model is a model based on text generation tasks, because it is relatively easy to explain the model's understanding of the code and the basis for predicting vulnerabilities, which helps developers understand the vulnerability mining results. This embodiment uses a pre-trained large language model as the initial model of the software security vulnerability intelligent mining model, which can improve the ability to mine software code security vulnerabilities.

[0082] In step S4, a sample data set is created according to the program dependency atlas, including:

[0083] S401. Assign vulnerability labels to the program dependency graphs in the program dependency graph set, and bind each vulnerability label to its corresponding program dependency graph; specifically, the vulnerability label is a label that does not contain a vulnerability. , and the existence of specified vulnerability types Tags For example, assigning labels to program dependency graphs with buffer overflow types ,Assign labels to non-vulnerable program dependency graphs .

[0084] S402. Create a sample data set based on each program dependency graph bound to a vulnerability label; wherein the sample data set includes a training set, a test set, and a validation set obtained by dividing each program dependency graph bound to a vulnerability label according to a preset ratio. Specifically, in this embodiment, the ratio of the training set, the test set, and the validation set is set to 7:2:1.

[0085] In this embodiment, the pre-trained large language model adopts the DeepSeek-Coder-6.7B-instruct large language model, and a classification head adopting a Dropout layer + Linear layer + Softmax layer network architecture is set on the top of the DeepSeek-Coder-6.7B-instruct large language model; the lightweight fine-tuning method adopts the P-Tuning fine-tuning method. It should be noted that the core idea of ​​the P-Tuning fine-tuning method is to insert trainable tokens at specific positions so that the model can better understand the requirements of downstream tasks.

[0086] It should be noted that the DeepSeek-Coder-6.7B-instruct large language model is a 6.7 billion parameter version of the open source pre-trained large language model released by the DeepSeek team. The pre-training data set of this model consists of 87% source code, 10% code-related English natural language corpus, and 3% code-independent Chinese natural language corpus, covering mainstream programming languages ​​such as Python and Java. It not only surpasses the 70 billion parameter Llama2 in many Chinese and English public evaluation lists, but also performs outstandingly in reasoning, mathematics, and programming capabilities. Based on this, this embodiment is sensitive to the syntax, semantics, and vulnerability forms of multiple high-level programming languages, which helps to improve the generalization ability of different programming languages.

[0087] Traditional code vulnerability mining methods based on large language models are either based only on prompt engineering without model training, such as the invention application with application number CN202311408265.0, or use full parameter fine-tuning methods, which have high requirements on hardware resources and slow model training time, such as the paper "Security Vulnerability Detection Using DeepLearning Natural Language Processing". Compared with the above-mentioned traditional fine-tuning technology, this embodiment only adds a classification head to the head of the DeepSeek-Coder-6.7B-instruct large language model, and uses the P-tuning fine-tuning method to train the pre-trained large language model. In this process, the model parameters are frozen, which can effectively reduce the size of model training parameters and computational complexity, accelerate the fine-tuning process, and maintain a high code vulnerability mining performance, so that the present invention has a wide range of application prospects in scenarios such as large-scale data sets and real-time processing.

[0088] Specifically, in this embodiment, the pre-trained large language model can be set to , whose hidden layer size is , the vocabulary size is , assuming the sample data set is ,in The input data consists of a series of discrete tokens, obtained from the program dependency graph. The sample data set is Input data, , which is the data in the sample data set The goal of training the pre-trained large language model using a lightweight fine-tuning method is to estimate the conditional probability of classification. .

[0089] In the process of training the pre-trained large language model using the sample data set and the P-tuning fine-tuning method, as shown in FIG. Figure 4 As shown, first, assume It is A discrete prompt is used to represent Convert to continuous prompt embedding When using the lightweight P-tuning method, each prompt can be described as a template .

[0090] ;

[0091] Then, an additional prompt encoder is used to implement a mapping function , for the prompt Mapping, embedding the trainable Mapping to the pre-trained large language model The input space Specifically, in this embodiment, a long short-term memory network and a multilayer perceptron (MLP) are used as prompt encoders to implement the following mapping:

[0092] ;

[0093] in, ;

[0094] Finally, the prompt embedding is iteratively updated by optimizing the loss function Complete the pre-trained large language model fine-tuning process.

[0095] In the process of training the pre-trained large language model using the lightweight fine-tuning method, this embodiment freezes the parameters in the pre-trained large language model based on the P-tuning fine-tuning method, and only updates the prompt embedding to achieve lightweight fine-tuning of the pre-trained large language model, which can greatly shorten the model training time and save hardware resource overhead.

[0096] S5. Receive the code to be tested, and perform vulnerability mining on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results. Specifically, in this embodiment, the code to be tested is subjected to vulnerability mining based on the software security vulnerability intelligent mining model to determine whether there are vulnerabilities in the code to be tested, and if there are vulnerabilities, the mined vulnerabilities are classified based on the CWE vulnerability library.

[0097] In this embodiment, the vulnerability mining result includes vulnerability label prediction results and specified vulnerability information; correspondingly, in step S5, vulnerability mining processing is performed on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results, including:

[0098] S501. Obtain a dependency graph of the program to be tested corresponding to the code to be tested; specifically, in the process of obtaining the dependency graph of the program to be tested corresponding to the code to be tested, the code to be tested is converted into a code to be tested middleware in advance, and then converted into a dependency graph of the program to be tested. This step can be carried out by processing the data in the vulnerability code data set to obtain the corresponding program dependency graph. The relevant steps are not repeated here.

[0099] S502. Input the dependency graph of the program to be tested into the intelligent mining model for software security vulnerabilities, so that the intelligent mining model for software security vulnerabilities can perform inference prediction on the dependency graph of the program to be tested and obtain vulnerability label prediction results; specifically, in this embodiment, the vulnerability label prediction results include all code snippets in the dependency graph of the program to be tested and their corresponding vulnerability labels, and the vulnerability labels include labels and tags One of them.

[0100] S503. Query the CWE vulnerability database for the specified vulnerability information that matches the vulnerability label prediction result. Specifically, in this embodiment, the specified vulnerability information includes a detailed description of the vulnerability, vulnerability impact, and its mitigation measures, etc., which are not limited here.

[0101] This embodiment can automatically, efficiently and accurately mine potential security vulnerabilities in software source code, and can obtain complete vulnerability mining results. Specifically, this embodiment uses a pre-trained large language model as the initial model of the software security vulnerability intelligent mining model, by abstractly representing the code data in the vulnerability code data set as a program dependency graph, and creating a sample data set based on the program dependency graph, and then using the sample data set to perform lightweight fine-tuning training on the pre-trained large language model, so as to realize the intelligent mining of software security vulnerabilities of the code to be tested based on the obtained software security vulnerability intelligent mining model. In this process, this embodiment uses the large language model's powerful understanding ability of natural language and code language to realize intelligent mining of software security vulnerabilities, and at the same time can better capture the dependency relationship, call relationship and logical flow in the code to be tested, which helps to identify potential vulnerabilities hidden behind the code, thereby improving the mining ability of this embodiment for potential security vulnerabilities in software code. In addition, traditional code vulnerability detection methods can only detect whether the code has vulnerabilities, but cannot classify the vulnerabilities and give vulnerability descriptions because they only perform binary classification training on the model or are limited by their neural network performance. However, before the vulnerability mining process is performed on the code to be tested, this embodiment collects open source vulnerability data to build a CWE vulnerability library. Therefore, when the software security vulnerability intelligent mining model is used to mine the code to be tested, the prediction results of the large language model are associated with the data in the CWE vulnerability library. When potential vulnerabilities are detected in the code to be tested, the vulnerabilities can be further identified based on the CWE vulnerability library to obtain vulnerability mining results including detailed descriptions of the vulnerabilities, vulnerability impacts, and mitigation measures. This can better discover security issues in the software code so that users and software manufacturers can repair them in a timely manner.

[0102] Embodiment 2:

[0103] This embodiment discloses a software security vulnerability intelligent mining system based on a large language model, which is used to implement the software security vulnerability intelligent mining method based on a large language model in Example 1; Figure 5 As shown, the software security vulnerability intelligent mining system based on the large language model includes:

[0104] A sample data set acquisition module is used to acquire a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set; it is also used to construct a program dependency graph based on the preprocessed vulnerability code data set, and create a sample data set based on the program dependency graph;

[0105] A model training module is connected to the sample data set acquisition module for acquiring a pre-trained large language model, and uses the sample data set to train the pre-trained large language model using a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model;

[0106] A CWE vulnerability library construction module is used to obtain CWE vulnerability data and construct a CWE vulnerability library based on the CWE vulnerability data;

[0107] The vulnerability mining module is communicatively connected with the model training module and the CWE vulnerability library construction module, and is used to receive the code to be tested, and perform vulnerability mining processing on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results.

[0108] It should be noted that the working process, working details and technical effects of the software security vulnerability intelligent mining system based on a large language model provided in this embodiment 2 can be found in embodiment 1 and will not be repeated here.

[0109] Embodiment 3:

[0110] Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. Figure 6 As shown, the electronic equipment includes:

[0111] a memory for storing computer program instructions; and,

[0112] A processor is used to execute the computer program instructions to complete the operation of the software security vulnerability intelligent mining method based on a large language model as described in any one of Example 1.

[0113] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0114] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the software security vulnerability intelligent mining method based on a large language model provided in Example 1 of the present application.

[0115] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0116] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0117] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.

[0118] The display screen 305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof.

[0119] The power supply 306 is used to supply power to various components in the electronic device.

[0120] Embodiment 4:

[0121] Based on any one of Examples 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for intelligent mining of software security vulnerabilities based on a large language model as described in any one of Example 1.

[0122] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent mining of software security vulnerabilities based on a large language model, characterized in that: include: Acquire a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set; Acquire CWE vulnerability data, and construct a CWE vulnerability library based on the CWE vulnerability data; Constructing a program dependency graph based on the preprocessed vulnerability code dataset; A sample data set is created according to the program dependency atlas, a pre-trained large language model is obtained, and the pre-trained large language model is trained using the sample data set and a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model; Receive the code to be tested, and perform vulnerability mining processing on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain a vulnerability mining result; The pre-processed vulnerability code data set includes a plurality of deduplicated code middlewares; Correspondingly, a program dependency graph is constructed based on the preprocessed vulnerability code dataset, including: Each deduplicated code middleware in the preprocessed vulnerability code data set is constructed as an abstract syntax tree, and a code basic block is extracted from each abstract syntax tree, and a control flow relationship between code basic blocks is extracted according to the control flow transfer statements in each abstract syntax tree; Taking the code basic blocks as nodes and the control flow relationships as control flow edges, a control flow graph corresponding to each deduplicated code middleware is constructed; Obtaining data flow dependencies of each node in the control flow graph, and using the data flow dependencies as data dependency edges, constructing a program dependency graph corresponding to each deduplicated code middleware based on the control flow graph, wherein each program dependency graph constitutes a program dependency graph set; After constructing the program dependency graph, the method further includes: Pruning the program dependency graph to obtain pruned program dependency graphs corresponding to each deduplicated code middleware, so as to construct a sample data set based on each pruned program dependency graph; The sample data set is created according to the program dependency atlas, including: Assign vulnerability labels to the program dependency graphs in the program dependency graph set respectively, and bind each vulnerability label to its corresponding program dependency graph; A sample data set is created based on each program dependency graph bound to a vulnerability label; wherein the sample data set includes a training set, a test set, and a validation set obtained by dividing each program dependency graph bound to a vulnerability label according to a preset ratio; The pre-trained large language model adopts the DeepSeek-Coder-6.7B-instruct large language model, and a classification head is arranged on the top of the DeepSeek-Coder-6.7B-instruct large language model; the lightweight fine-tuning method adopts the P-Tuning fine-tuning method; The vulnerability mining result includes vulnerability label prediction results and specified vulnerability information; correspondingly, vulnerability mining processing is performed on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results, including: Obtaining a dependency graph of a program under test corresponding to the code under test; Inputting the dependency graph of the program to be tested into the software security vulnerability intelligent mining model, so that the software security vulnerability intelligent mining model performs inference prediction on the dependency graph of the program to be tested to obtain vulnerability label prediction results; Query the CWE vulnerability database for specified vulnerability information that matches the vulnerability label prediction result.

2. According to claim 1, a method for intelligently mining software security vulnerabilities based on a large language model is characterized in that: The vulnerability code data set includes a plurality of code middlewares; correspondingly, the vulnerability code data set is preprocessed to obtain a preprocessed vulnerability code data set, including: Based on the MinHash algorithm, obtain the code snippet of each code middleware in the vulnerability code data set, and construct a code snippet set based on the code snippet of each code middleware, and obtain the MinHash signature of each code snippet in the code snippet set; Based on the LSH algorithm, the MinHash signature of each code snippet in the code snippet set is used as a high-dimensional vector to construct an LSH hash table; The MinHash signatures in the same hash table position in the LSH hash table are calculated similarity pair by pair. When the calculated similarity is higher than a preset threshold, a representative signature is retained and the remaining similar MinHash signatures are removed to obtain the deduplicated MinHash signatures. The deduplicated MinHash signatures are then mapped back to the corresponding code snippets to construct a preprocessed vulnerability code dataset.

3. A software security vulnerability intelligent mining system based on a large language model, characterized in that: Used to implement the software security vulnerability intelligent mining method based on a large language model as described in any one of claims 1 to 2; The software security vulnerability intelligent mining system based on the large language model includes: A sample data set acquisition module is used to acquire a vulnerability code data set, and preprocess the vulnerability code data set to obtain a preprocessed vulnerability code data set; it is also used to construct a program dependency graph based on the preprocessed vulnerability code data set, and create a sample data set based on the program dependency graph; A model training module is connected to the sample data set acquisition module for acquiring a pre-trained large language model, and uses the sample data set to train the pre-trained large language model using a lightweight fine-tuning method to obtain a software security vulnerability intelligent mining model; A CWE vulnerability library construction module is used to obtain CWE vulnerability data and construct a CWE vulnerability library based on the CWE vulnerability data; The vulnerability mining module is communicatively connected with the model training module and the CWE vulnerability library construction module, and is used to receive the code to be tested, and perform vulnerability mining processing on the code to be tested based on the software security vulnerability intelligent mining model and the CWE vulnerability library to obtain vulnerability mining results.

Images