A vulnerability detection method based on a large code change model

Abstract

The present invention proposes a vulnerability detection method based on a large model of code changes. The method comprises the following steps: preprocessing the data set, deleting duplicate samples and comments, retaining only the code text, tokenizing the code text and removing duplicate tokens. Then, traversing the code text without vulnerability features, calculating the Jaccard similarity with the target code, and selecting a data sample pair with the highest similarity to the code text without vulnerability features. The code text without vulnerability features is regarded as a pre-change fragment, and the target code is regarded as a post-change fragment. The difference between the codes is extracted and marked as an added or deleted code fragment. The model is initialized using the pre-trained weights of the large model CCT5 based on code changes, the model structure is adjusted, the key parameters are set, the model is fine-tuned on the training set and the parameters are updated for detecting code vulnerabilities. A section of code text is input into the CCT5 model that performs best in the fine-tuning task, and the model outputs a prediction value of whether it has a vulnerability.

Description

Technical Field

[0001] The present invention relates to an influencing factor analysis method, and in particular to a vulnerability detection method based on a large code change model. Background Art

[0002] With the widespread use of information technology in modern society, software vulnerabilities have caused significant damage to individuals and businesses. Many internet users' personal information has been compromised due to security breaches. According to a 2021 IBM report, the average cost of a data breach to a targeted company is $4.24 million. The number of disclosed security vulnerabilities has also been increasing year by year. According to a report released by the security database website CVE Details, 20,153 security vulnerabilities were reported in 2021, 25,082 in 2022, 29,066 in 2023, and 35,062 as of October 2024. Over the past decade, over 19.1% of reported vulnerabilities have received a CVSS score exceeding 9.0. In addition to the increasing number of vulnerabilities, the types of vulnerabilities have also become more complex and diverse, posing a significant threat to the normal and secure operation of computer systems.

[0003] Traditional vulnerability detection techniques analyze the lexical, syntactic, and semantic aspects of a target program without executing it. These techniques, combined with data flow and control flow information, leverage techniques such as type inference, security rule checking, and model checking to discover potential vulnerabilities within the program. With the booming artificial intelligence industry, many methods based on deep neural networks have achieved significant success in fields such as image recognition and natural language processing. The deep learning-based vulnerability detection workflow primarily consists of two key steps: the training phase and the detection phase. From a code representation perspective, existing vulnerability detection methods directly encode individual code segments and then use deep learning techniques to predict whether the code contains a vulnerability. However, these methods are less effective when processing long code segments, as many neural models have input length limitations, which restrict their ability to process long vulnerable code snippets. Furthermore, these methods struggle to distinguish vulnerability-related information from a large amount of irrelevant information, limiting their effectiveness in capturing vulnerability signatures. Summary of the Invention

[0004] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to more accurately predict whether the code has a vulnerability.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a vulnerability detection method based on a large code change model, including component one for constructing differences between similar sample pairs and component two for detecting whether a vulnerability exists.

[0006] Component 1 includes the following steps:

[0007] S101: Obtain labeled data samples as the original data set D = {d1, d2, d3, ..., d n}, where d n represents the nth original data sample. The label is whether the original data sample has a vulnerability. The original data sample is preprocessed, including deleting duplicate data samples and removing comments in the code. Only the code text is retained to obtain a new data set D″={d′1,d′2,d′3,…,d′ m}, where d′ m represents the mth code text, m≤n, and the labels of the samples in D″ and the corresponding samples in D remain unchanged.

[0008] Specifically, duplicate samples are deleted to ensure the uniqueness of samples in the dataset. The original dataset is D = {d1, d2, d3, ..., d n}, the dataset after deleting duplicate samples is recorded as D′. The code text in each data sample in D′ is parsed, the comments in the code are removed, and only the code text is retained. The processed dataset is expressed as D″={d′1,d′2,d′3,…,d′ m}, where m≤n.

[0009] S102: The i-th code text d′ in D″ i Tokenize to get token list T i ={t1,t2,t3,…,t k}, where t k Indicates T i The kth element in T i Perform deduplication operation so that T i Only unique tokens are retained, and the token set after deduplication is recorded as in Represents T′ i The p-th element in , p≤k, ensures that each code text only retains a unique token list.

[0010] Specifically, the code text d′ i Tokenize d′ i Split into token list T i ={t1,t2,t3,…,t k}, where each t j Represents a code token (such as a variable, operator, keyword, etc.).

[0011] For each token list T i Perform deduplication operation so that Ti Only unique tokens are retained. The token set after deduplication is Where p≤k.

[0012] S103: For each T′ i , traverse all the samples T′ without loopholes in D″ j , calculate T′ i and T′ j The Jaccard similarity between them will find the most similar vulnerability-free code text and T′ i Composition sample pairs

[0013] S104: Considered as the code snippet “before the change”, T′ i Considered as "changed" code snippet, calculate Relative to T′ i Added code snippet Δ add , deleted code snippet Δ delete and the difference set

[0014] Specifically, let the code snippet "before the change" be The "changed" code snippet is T after =T′ i .calculate Relative to T′ i Added and deleted fragments Δ add and Δ delete :

[0015]

[0016] Get the difference code snippet set Δ=(Δ add ,Δ delete ).

[0017] S105: Construct the final data set D″′={d″1,d″2,d″3,…,d″ m},in Among them L m Indicates label, L m The labels of the corresponding original data samples are the same.

[0018] Component 2 includes the following steps:

[0019] S106: As input to fine-tune the large model CCT5 based on code changes to optimize its parameters, θ prerepresents the pre-trained weights of CCT5, connects the [CLS] labeled representation of the last layer of the CCT5 encoder to the linear classifier, and uses θ pre Initialize CCT5 parameters.

[0020] S107: Divide D″ into a training set and a validation set, use the training set to fine-tune CCT5, randomly select a number of training samples from the training set to form a batch, calculate the loss after completing the training of a batch, and reversely update the model parameters θ of CCT5 based on the loss.

[0021] Specifically, the model parameters θ are updated through back propagation:

[0022]

[0023] Where α is the learning rate, is the loss function Gradient with respect to the model parameters θ.

[0024] S108: Input the test samples in the validation set into CCT5 after updating θ in S107, and calculate the F1 score. If the F1 score reaches the highest value and does not change, save the current model parameters as θ best , and record the current CCT5 as the optimal CCT5, otherwise return to S107;

[0025] S109: For a piece of code text X test , X test Input optimal CCT5, optimal CCT5 output X test Prediction value y of whether there is a vulnerability test ∈{0,1}.

[0026] Furthermore, in S103, the Jaccard similarity is calculated as follows:

[0027]

[0028] where |T i ′∩T j ′| represents the intersection size of two token sets, |T i ′∪T j ′| represents the union size of two token sets.

[0029] Furthermore, the step S106 includes the following steps: Before entering CCT5, and Use separately <add>and <del>Mark connections;

[0030]

[0031] Among them, X add and X del Respectively represent the added items and deleted items after marking; x n and y n They represent the nth added item and the nth deleted item respectively.

[0032] To adapt to the vulnerability detection task, the [CLS] tag output by the last layer of the encoder is represented by h CLS As a global representation, h CLS Connect to a linear classifier f linear , defined as:

[0033] h CLS =Encoder([X add ;X del ]) CLS

[0034] y=f linear (h CLS )

[0035] Among them, y∈(0,1) represents the final prediction result of the model.

[0036] Furthermore, in S107, the process of calculating the loss is as follows:

[0037] Define the loss function for:

[0038]

[0039] Among them, B represents a set of training samples in a batch, is the predicted probability of the i-th training sample (between 0 and 1), y i represents the true label (0 or 1) of the i-th sample.

[0040] Furthermore, in S108, at the end of each training step, the model is verified and the F1 score is calculated to evaluate its performance in the vulnerability detection task. The process of calculating the F1 score is as follows:

[0041]

[0042] Among them, Precision refers to the precision rate, which refers to the proportion of samples that are actually positive among all samples predicted by the model to be positive; Recall refers to the recall rate, which refers to the proportion of samples that are correctly predicted to be positive by the model among all samples that are actually positive.

[0043] Compared with the prior art, the present invention has at least the following advantages:

[0044] 1. This invention solves the technical problem by constructing two main components: constructing the differences between similar code pairs and detecting whether there are vulnerabilities. It aims to identify potential security vulnerabilities by analyzing the change process and semantic characteristics between vulnerability-free and vulnerability-featured codes.

[0045] Component 1 preprocesses the code data, removing duplicates and comments and converting the code into tokens. It then calculates the Jaccard similarity between the target code and the vulnerability-free code to identify the most similar code pairs. It also pinpoints the changes between the two versions, particularly newly added or deleted code segments, allowing the model to focus on code changes that may harbor vulnerability characteristics.

[0046] Component 2 utilizes a large model (CCT5) based on code changes, which is further fine-tuned to make it suitable for vulnerability detection tasks. During this process, the model is optimized using a training set and tested on a validation set. Ultimately, the optimal model parameter settings are determined to ensure optimal performance in real-world vulnerability detection scenarios.

[0047] 2. Performance Improvement: Experimental results show that the proposed method has significant improvements in key performance indicators such as Precision, Recall, and F1-score compared to other methods. The F1-score reaches 55.62%, indicating that the model achieves a better balance between precision and recall.

[0048] 3. Effectively identify vulnerabilities: The present invention performs particularly well in correctly identifying vulnerabilities, which shows that it can effectively identify potential security vulnerabilities from the code.

[0049] 4. Practical application value: The method of the present invention not only has superior performance in theory, but can also be applied in actual work scenarios, helping developers detect and prevent security vulnerabilities during the software development process and reduce the risks of system failures and data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a simplified flow chart of the method of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in further detail below.

[0052] See also Figure 1 This paper proposes a vulnerability detection method based on a large code change model. Starting with the semantic representation of code edit sequences, this method identifies modified code snippets with vulnerability characteristics by looking for differences between them and non-vulnerable code snippets. These modified code snippets are then encoded using a pre-trained model for classification prediction. This method consists of a first component for constructing differences between similar sample pairs and a second component for detecting vulnerabilities in the code.

[0053] Specifically, component one is responsible for constructing the differences between similar sample pairs. First, it obtains labeled data samples and preprocesses them, removing duplicate samples and annotations while retaining the code text. Then, it tokenizes the code text and removes duplicate tokens, ensuring that each code text contains only a unique list of tokens. Next, it calculates the Jaccard similarity between the target code and the vulnerability-free code in the training set, and selects the most similar vulnerability-free code text to form a sample pair. Finally, it extracts the differences between the two code segments, including added and deleted code snippets, and marks these differences. Component two is used to detect vulnerabilities. First, it uses the differences as input, initializes the CCT5 model using pre-trained weights, adjusts the model structure, and sets key parameters. Next, it divides the dataset into training, validation, and test sets. It uses the training set to fine-tune CCT5 and calculates the loss function to reversely update the model parameters. The model performance is evaluated on the validation set, and the optimal model parameters are selected by calculating the F1 score. Finally, the code is input into the optimal CCT5 model, which outputs a prediction value indicating whether it has a vulnerability, thereby achieving vulnerability detection.

[0054] A vulnerability detection method based on a large code change model includes a component one for constructing differences between similar sample pairs and a component two for detecting whether a vulnerability exists.

[0055] Component 1 includes the following steps:

[0056] S101: Obtain labeled data samples as the original data set D = {d1, d2, d3, ..., d n }, where d n represents the nth original data sample. The label is whether the original data sample has a vulnerability. The original data sample is preprocessed, including deleting duplicate data samples and removing comments in the code. Only the code text is retained to obtain a new data set D″={d′1,d′2,d′3,…,d′ m }, where d′ m represents the mth code text, m≤n, and the labels of the samples in D″ and the corresponding samples in D remain unchanged.

[0057] Specifically, duplicate samples are deleted to ensure the uniqueness of samples in the dataset. The original dataset is D = {d1, d2, d3, ..., d n}, the dataset after deleting duplicate samples is recorded as D′. The code text in each data sample in D′ is parsed, the comments in the code are removed, and only the code text is retained. The processed dataset is expressed as D″={d′1,d′2,d′3,…,d′ m }, where m≤n.

[0058] S102: The i-th code text d′ in D″ i Tokenize to get token list T i ={t1,t2,t3,…,t k }, where t k Indicates T i The kth element in T i Perform deduplication operation so that T i Only unique tokens are retained, and the token set after deduplication is recorded as in Represents T′ i The p-th element in , p≤k, ensures that each code text only retains a unique token list.

[0059] Specifically, the code text d′ i Tokenize d′ i Split into token list T i ={t1,t2,t3,…,t k }, where each t j Represents a code token (such as a variable, operator, keyword, etc.).

[0060] For each token list T i Perform deduplication operation so that T i Only unique tokens are retained. The token set after deduplication is Where p≤k.

[0061] S103: For each T′ i , traverse all the samples T′ without loopholes in D″ j , calculate T′ i and T′ j The Jaccard similarity between them will find the most similar vulnerability-free code text and T′ i Composition sample pairs

[0062] S104: The code snippet considered as "before change", T′ i Considered as a "changed" code snippet, calculate Relative to T′ i Added code snippet Δ add , deleted code snippet Δ delete and the difference set

[0063] Specifically, let the "before change" code snippet be The "after change" code snippet is T after =T′ i .calculate Relative to T′ i Added and deleted fragments Δ add and Δ delete :

[0064]

[0065] Get the difference code snippet set Δ=(Δ add ,Δ delete ).

[0066] S105: Construct the final data set D″′={d″1,d″2,d″3,…,d″ m },in Among them L m Indicates label, L m The labels of the corresponding original data samples are the same.

[0067] Component 2 includes the following steps:

[0068] S106: As input to fine-tune the large model CCT5 based on code changes to optimize its parameters, θ pre represents the pre-trained weights of CCT5, connects the [CLS] labeled representation of the last layer of the CCT5 encoder to the linear classifier, and uses θ pre Initialize CCT5 parameters.

[0069] S107: Divide D″ into a training set and a validation set, use the training set to fine-tune CCT5, randomly select a number of training samples from the training set to form a batch, calculate the loss after completing the training of a batch, and reversely update the model parameters θ of CCT5 based on the loss.

[0070] Specifically, the model parameters θ are updated through back propagation:

[0071]

[0072] Where α is the learning rate, is the loss function Gradient with respect to the model parameters θ.

[0073] S108: Input the test samples in the validation set into CCT5 after updating θ in S107, and calculate the F1 score. If the F1 score reaches the highest value and does not change, save the current model parameters as θ best , and record the current CCT5 as the optimal CCT5, otherwise return to S107;

[0074] S109: For a piece of code text X test , X test Input optimal CCT5, optimal CCT5 output X test Prediction value y of whether there is a vulnerability test ∈{0,1}.

[0075] Furthermore, in S103, the Jaccard similarity is calculated as follows:

[0076]

[0077] where |T i ′∩T j ′| represents the intersection size of two token sets, |T i ′∪T j ′| represents the union size of two token sets.

[0078] Furthermore, the step S106 includes the following steps: Before entering CCT5, and Use separately <add>and <del>Mark connections;

[0079]

[0080] Among them, X add and X del Respectively represent the added items and deleted items after marking; x n and y n They represent the nth added item and the nth deleted item respectively.

[0081] To adapt to the vulnerability detection task, the [CLS] tag output by the last layer of the encoder is represented by h CLS As a global representation, h CLS Connect to a linear classifier f linear , defined as:

[0082] h CLS =Encoder([X add ;X del ]) CLS

[0083] y=f linear (h CLS )

[0084] Among them, y∈(0,1) represents the final prediction result of the model.

[0085] Furthermore, in S107, the process of calculating the loss is as follows:

[0086] Define the loss function for:

[0087]

[0088] Among them, B represents a set of training samples in a batch, is the predicted probability of the i-th training sample (between 0 and 1), y i represents the true label (0 or 1) of the i-th sample.

[0089] Furthermore, in S108, at the end of each training step, the model is verified and the F1 score is calculated to evaluate its performance in the vulnerability detection task. The process of calculating the F1 score is as follows:

[0090]

[0091] Among them, Precision refers to the precision rate, which refers to the proportion of samples that are actually positive among all samples predicted by the model to be positive; Recall refers to the recall rate, which refers to the proportion of samples that are correctly predicted to be positive by the model among all samples that are actually positive.

[0092] Experiment and analysis

[0093] To verify the effectiveness of this method in vulnerability detection, we used a constructed merged dataset. This dataset consists of three high-quality vulnerability datasets commonly used in vulnerability detection tasks. All of them are based on actual C / C++ vulnerabilities in open source projects. Table 1 lists the data distribution of the training set, validation set, and test set used in this paper.

[0094] Table 1 Dataset distribution statistics

[0095]

[0096] To improve the model's performance in code vulnerability detection tasks, we pre-trained and fine-tuned the model, thoroughly pre-processing and extracting features from the data samples, and initialized the model using pre-trained weights. To ensure stable training and optimization on different datasets, the key parameters set for this pre-training and fine-tuning task are shown in Table 2.

[0097] Table 2 Key parameters

[0098]

[0099] After model training, using optimal parameters, the performance of this method on the test dataset is shown in Table 3. Experimental results show that this method achieves a precision of 47.37%, a recall of 67.36%, and an F1-score of 55.62%. Compared with other methods, this method improves on all metrics to varying degrees, particularly in correctly identifying vulnerabilities, while maintaining a good balance between precision and recall.

[0100] Table 3 Performance comparison of different methods

[0101]

[0102] The vulnerability detection method proposed in this paper, based on a large code change model, demonstrates remarkable performance in vulnerability detection and can effectively identify vulnerabilities in code. Furthermore, this method can be applied in real-world scenarios to help detect security vulnerabilities in software development, prevent potential security risks, and avoid system failures or data leaks caused by vulnerabilities.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.< / del> < / add> < / del> < / add>

Claims

1. A vulnerability detection method based on a large code change model, characterized by: It includes component one for constructing differences between similar sample pairs and component two for detecting whether there are vulnerabilities; Component 1 includes the following steps: S101: Obtain labeled data samples as the original data set D = {d1, d2, d3, ..., d n }, where d n represents the nth original data sample. The label is whether the original data sample has a vulnerability. The original data sample is preprocessed, including deleting duplicate data samples and removing comments in the code. Only the code text is retained to obtain a new data set D″={d′ i , d′2, d′3, ..., d′ m }, where d′ m represents the mth code text, m≤n, and the labels of the samples in D″ and the corresponding samples in D remain unchanged; S102: The i-th code text d′ in D″ i Tokenize to get token list T i ={t1, t2, t3, ..., t k }, where t k Indicates T i The kth element in T i Perform deduplication operation so that T i Only unique tokens are retained, and the token set after deduplication is recorded as ,in Represents T′ i The p-th element in , p≤k, ensures that each code text only retains a unique token list; S103: For each T′ i , traverse all the samples T′ without loopholes in D″ j , calculate T′ i and T′ j The Jaccard similarity between them will find the most similar vulnerability-free code text T′ j* and T′ i Composition sample pair (T ′ i , T′ j* ); S104: T' j* The code snippet considered as "before change", T′ i Consider the code snippet as "after the change" and calculate T' j* Relative to T′ i Added code snippet Δ add , deleted code snippet Δ delete and the difference set S105: Construct the final data set D″′={d″1, d″2, d″3, ..., d″ m }, where d″ m =((T′ m , T′ n* ), Δ m , L m ), where L m Indicates label, L m The labels of the corresponding original data samples are the same; Component 2 includes the following steps: S106: As input to fine-tune the large model CCT5 based on code changes to optimize its parameters, θ pre represents the pre-trained weights of CCT5, connects the [CLS] labeled representation of the last layer of the CCT5 encoder to the linear classifier, and uses θ pre Initialize CCT5 parameters; S107: Divide D″ into a training set and a validation set, use the training set to fine-tune CCT5, randomly select a number of training samples from the training set to form a batch, calculate the loss after completing the training of a batch, and reversely update the model parameters θ of CCT5 based on the loss; S108: Input the test samples in the validation set into CCT5 after updating θ in S107, and calculate the F1 score. If the F1 score reaches the highest value and does not change, save the current model parameters as θ best , and record the current CCT5 as the optimal CCT5, otherwise return to S107; S109: For a piece of code text X test , X test Input optimal CCT5, optimal CCT5 output X test Prediction value y of whether there is a vulnerability test ∈{0,1}.

2. The vulnerability detection method based on a large code change model according to claim 1, characterized in that: In S103, the Jaccard similarity calculation process is as follows:

3. The vulnerability detection method based on a large code change model according to claim 2, characterized in that: The S106 includes the following steps: Before entering CCT5, and Use separately <add>and <del> Mark connections;< / del> < / add> Among them, X add and X del Respectively represent the added items and deleted items after marking; x n and y n They represent the nth added item and the nth deleted item respectively; The [CLS] tag output by the last layer of the encoder is represented as h CLS As a global representation, h CLS Connect to a linear classifier f linear , defined as: h CLS =Encoder([X add ;X del ]) CLS y=f linear (h CLS ) Among them, y∈(0,1) represents the final prediction result of the model.

4. The vulnerability detection method based on a large code change model according to claim 3, characterized in that: In S107, the process of calculating the loss is as follows: Define the loss function for: Among them, B represents a set of training samples in a batch, is the predicted probability y of the i-th training sample i represents the true label of the i-th sample.

5. The vulnerability detection method based on a large code change model according to claim 4, characterized in that: In S108, the process of calculating the F1 score is as follows: Among them, Precision refers to the precision rate, which refers to the proportion of samples that are actually positive among all samples predicted by the model to be positive; Recall refers to the recall rate, which refers to the proportion of samples that are correctly predicted to be positive by the model among all samples that are actually positive.

Images