Timely defect prediction method based on comparative learning and feature fusion
By introducing contrast learning and feature fusion in the semantic feature stage, the problem of insufficient accuracy in the existing technology when dealing with similar code changes is solved, and higher defect detection accuracy and robustness are achieved, and the model is enhanced to complex code Adaptability to the scene.
Patent Information
- Application Number
- CN202411489722.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-18
AI Technical Summary
Existing defect prediction methods are insufficient in handling similar code changes and are vulnerable to noise labels, making it difficult to effectively distinguish vulnerable code from non-vulnerable code.
Introducing contrast learning in the semantic feature stage, by maximizing the similarity between positive samples and minimizing the similarity between positive and negative samples, the model's ability to distinguish similarity between similar code changes is enhanced, and combining feature fusion to fuse semantic features with experts Features, optimize model architecture.
It improves the accuracy and robustness of the model in timely defect detection, can more effectively capture subtle differences between codes, improves the ability to adapt to complex code scenarios, and reduces the potential safety hazards and failure risks of the system.
Smart Images
Figure CN119473849A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a timely defect prediction method based on contrastive learning and feature fusion, aiming to improve the accuracy of defect detection in source code. Background Art
[0002] With the increasing complexity and scale of modern software development, the detection of software defects has become crucial. Most of the current defect prediction methods rely on manually designed features or traditional machine learning models. These methods usually predict defects by extracting manual features related to code changes (such as the number of modified files, developer experience, etc.). However, these methods often ignore the semantic information contained in the code, resulting in insufficient prediction accuracy.
[0003] In recent years, with the development of deep learning technology, especially pre-trained language models such as CodeBERT, they have begun to show their potential in the direction of timely defect prediction. However, fine-tuning these models using cross-entropy loss brings significant challenges, including limited generalization and susceptibility to noisy labels. These problems become particularly prominent when vulnerable code is very similar to non-vulnerable code, making it difficult for deep learning models to accurately distinguish.
[0004] How to solve the above problems has become the key of the present invention. By introducing contrastive learning in the semantic feature stage, the present invention can not only solve the model's lack of ability to distinguish similar code changes, but also alleviate the problem of noisy labels. The present invention combines contrastive learning with feature fusion to make up for the shortcomings of the existing technology, thereby effectively improving the performance of the model in the task of timely defect detection. Summary of the invention
[0005] The purpose of the present invention is to provide a timely defect prediction method based on contrastive learning and feature fusion, aiming to more effectively detect potential defects in code and improve the accuracy and robustness of detection.
[0006] The idea of the present invention is: the present invention proposes a timely defect prediction method based on contrastive learning and feature fusion, which calculates the similarity difference between positive and negative samples through contrastive learning, generates contrast loss, and increases the model's ability to distinguish similar codes. It combines the expert features and semantic features of the code through feature fusion to further enhance the model's ability to distinguish code defects. It enables the model to instantly predict potential defects when the code is submitted, reducing the potential safety hazards and failure risks of the system.
[0007] The present invention is achieved by the following measures: a timely defect prediction method based on contrast learning and feature fusion, which includes the following steps:
[0008] 1.1: Preprocess and split the data set into training set D1 and test set D2;
[0009] 1.2: Extract semantic features from the submission information and modification descriptions of the training set D1 divided in step 1.1;
[0010] 1.3: Apply contrastive learning and define positive samples z p and negative samples z n , by maximizing the similarity between positive samples and minimizing the similarity between positive samples and negative samples, the model's ability to distinguish similar code changes is enhanced;
[0011] 1.4: Calculate the positive sample z defined in step 1.3 p and z of negative samples n The contrast loss value of is obtained to obtain the total contrast loss, which is used to optimize the model;
[0012] 1.5: In the feature fusion stage, the semantic features extracted in step 1.2 are fused with the expert features, and the model architecture is optimized by fine-tuning the fully connected layer and activation function;
[0013] 1.6: Use the trained model to predict defects on the data in the test set D2 and output the prediction results.
[0014] Extracting semantic features from the submission information and modification descriptions of the training set D1 divided in step 1.1 in step 1.2 specifically includes the following steps:
[0015] 2.1: Use the CodeBERT model to convert each code snippet in the training set D1 into a feature vector. By inputting the context information of the code snippet, the semantic information in the code is captured and a feature vector of multiple dimensions is generated to represent the semantic content and structure of the code.
[0016] In step 1.3, contrastive learning is applied to define the positive sample z p and negative samples z n , by maximizing the similarity between positive samples and minimizing the similarity between positive samples and negative samples, the model's ability to distinguish similar code changes is enhanced. Specifically, the following steps are included:
[0017] 3.1: Consider code changes with the same defect labels in training set D1 as positive samples z p , consider code changes with different defect labels as negative samples z n .
[0018] In step 1.4, the positive sample z defined in step 1.3 is calculated p and z of negative samples nThe contrast loss value is obtained to obtain the total contrast loss, which is used to optimize the model. Specifically, the following steps are included:
[0019] 4.1: First, sample z i The embedding representation and positive sample z p The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z p );
[0020] 4.2: Then the sample z i The embedding representation and negative sample z n The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z n );
[0021] 4.3: Bring the similarity measure of positive samples and negative samples into the contrast loss function, and finally calculate the contrast loss L contrast , the specific formula is as follows:
[0022]
[0023] where z i represents the embedding of sample i, z p Indicates that z i The embedding of positive samples with the same label, z n Indicates that z i Embeddings of negative samples with different labels, sim represents the similarity measure between samples.
[0024] In the feature fusion stage in step 1.5, the semantic features extracted in step 1.2 are fused with the expert features, and the model architecture is optimized by fine-tuning the fully connected layer and the activation function, which specifically includes the following steps:
[0025] 5.1: Standardize the semantic features and expert features extracted from step 1.2 so that the two types of features can be fused at the same scale;
[0026] 5.2: Concatenate the expanded semantic features and expert features to generate a comprehensive feature representation that integrates the information of semantic and expert features;
[0027] 5.3: Input the concatenated comprehensive features into the multi-layer fully connected layer, and use the activation function to perform nonlinear transformation on the features;
[0028] 5.4: Based on the fully connected layer, the back propagation algorithm is used to update the weights of the model and adjust the parameters of the fully connected layer and activation function to maximize the predictive performance of the model.
[0029] In step 1.6, the trained model is used to predict defects on the data in the test set D2 and the prediction results are output.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] 1. The present invention introduces contrastive learning and feature fusion to solve the shortcomings of current pre-trained models in distinguishing similar codes and capturing contextual information, enhances the model's adaptability to complex code scenarios, and further improves the accuracy of timely defect prediction.
[0032] 2. The present invention solves the difficulty of existing pre-trained models in distinguishing similar code changes by introducing contrastive learning. Although traditional pre-trained models, such as CodeBERT, can extract semantic features of codes, they often perform poorly when dealing with code changes that are highly similar in semantics or structure, and are prone to misclassifying code changes with different labels as the same. The present invention introduces a contrastive learning method in the semantic feature extraction stage, which maximizes the similarity between positive samples and minimizes the similarity between positive and negative samples, thereby enhancing the model's ability to distinguish when dealing with similar code changes. By constructing a contrastive loss function, the model can more effectively capture subtle differences between codes, especially those code changes that are highly similar in semantics but different in function or behavior, greatly improving the accuracy of the model in defect detection, especially when dealing with complex code modifications.
[0033] 3. The present invention introduces feature fusion to solve the shortcomings of single semantic features or expert features in capturing contextual information. Traditional methods usually rely on a single feature extraction method and cannot capture multi-dimensional information of the code at the same time, resulting in insufficient contextual understanding. The present invention combines expert features (such as the degree of diffusion of code changes, the number of file modifications, developer experience, etc.) with semantic features extracted by CodeBERT, and fuses these two types of features in the feature fusion stage. This feature fusion method enables the model to understand the structural information and semantic information of code changes at the same time, improves the model's ability to understand the code context, and through further optimization of multiple fully connected layers and activation functions, feature fusion enhances the model's adaptability to complex code scenarios and further improves the accuracy of timely defect prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0035] Figure 1 A system framework diagram of a timely defect prediction method based on contrastive learning and feature fusion provided by the present invention. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0037] Example 1
[0038] See also Figure 1 As shown, this embodiment provides a timely defect prediction method based on contrastive learning and feature fusion, which specifically includes the following steps:
[0039] (1-1) Preprocess and split the data set into training set D1 and test set D2;
[0040] (1-2) extracting semantic features from the submission information and modification descriptions of the training set D1 divided in step (1-1);
[0041] (1-3) Apply contrastive learning and define positive sample z p and negative samples z n , by maximizing the similarity between positive samples and minimizing the similarity between positive samples and negative samples, the model's ability to distinguish similar code changes is enhanced;
[0042] (1-4) Calculate the positive sample z defined in step (1-3) p and z of negative samples n The contrast loss value of is used to obtain the total contrast loss, which is used to optimize the model.
[0043] (1-5) In the feature fusion stage, the semantic features extracted in step (1-2) are fused with the expert features, and the model architecture is optimized by fine-tuning the fully connected layer and activation function;
[0044] (1-6) Use the trained model to predict defects on the data in the test set D2 and output the prediction results.
[0045] The step (1-2) of extracting semantic features from the submission information and modification description of the training set D1 divided in step 1.1 specifically includes the following steps:
[0046] (2-1) Use the CodeBERT model to convert each code snippet in the training set D1 into a feature vector. By inputting the context information of the code snippet, the semantic information in the code is captured and a feature vector of multiple dimensions is generated to represent the semantic content and structure of the code.
[0047] In the steps (1-3), contrastive learning is applied to define the positive sample z p and negative samples z n, by maximizing the similarity between positive samples and minimizing the similarity between positive samples and negative samples, the model's ability to distinguish similar code changes is enhanced, including the following steps:
[0048] (3-1) The code changes with the same defect labels in the training set D1 are regarded as positive samples z p , consider code changes with different defect labels as negative samples z n .
[0049] In the step (1-4), the positive sample z defined in the step (1-3) is calculated p and z of negative samples n The contrast loss value is obtained to obtain the total contrast loss, which is used to optimize the model. Specifically, the following steps are included:
[0050] (4-1) First, sample z i The embedding representation and positive sample z p The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z p );
[0051] (4-2) Then the sample z i The embedding representation and negative sample z n The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z n );
[0052] (4-3) The similarity measure of positive samples and negative samples is brought into the contrast loss function, and finally the contrast loss L is calculated contrast , the specific formula is as follows:
[0053]
[0054] where z i represents the embedding of sample i, z p Indicates that z i The embedding of positive samples with the same label, z n Indicates that z i Embeddings of negative samples with different labels, sim represents the similarity measure between samples.
[0055] In the feature fusion stage of step (1-5), the semantic features extracted in step (1-2) are fused with the expert features, and the model architecture is optimized by fine-tuning the fully connected layer and the activation function, which specifically includes the following steps:
[0056] (5-1) The semantic features and expert features extracted from step (1-2) are standardized so that the two types of features can be fused at the same scale;
[0057] (5-2) Concatenate the expanded semantic features and expert features to generate a comprehensive feature representation that integrates the information of semantic and expert features;
[0058] (5-3) Input the concatenated comprehensive features into the multi-layer fully connected layer, and use the activation function to perform nonlinear transformation on the features;
[0059] (5-4) Based on the fully connected layer, the back propagation algorithm is used to update the model weights and adjust the parameters of the fully connected layer and activation function to maximize the predictive performance of the model.
[0060] In step 1.6, the trained model is used to predict defects on the data in the test set D2 and the prediction results are output.
[0061] (7) The proposed method and the existing timely defect prediction methods were evaluated on the same dataset, and five performance indicators from the timely defect prediction research field (i.e., F1-score, AUC, Recall@20% Effort, Effort@20% Recall, and POpt) were used to automatically evaluate the quality of the model.
[0062] Table 1 Comparison of results of the method of the present invention and other methods
[0063]
[0064]
[0065] Experiments show that the timely defect prediction method based on contrastive learning and feature fusion proposed in this embodiment can make more accurate timely defect predictions than the baseline method. Specifically, the method of this embodiment combines expert features and semantic features extracted by CodeBERT, introduces contrastive learning in the semantic feature extraction stage, and constructs a contrast loss function. The expert features and semantic features are fused through feature fusion, so that the model can better distinguish similar code changes, further improve the accuracy of timely defect prediction, and surpass these baseline methods; among them, for F1-scor, the method of this embodiment can at least relatively improve the performance by 13.9%; for AUC, the method of this embodiment can at least relatively improve the performance by 0.65%; for R@20%E, the method of this embodiment can at least relatively improve the performance by 7.29%; for E@20%R, the method of this embodiment achieves the same performance as the current optimal method; for Popt, the method of this embodiment can at least relatively improve the performance by 1.86%; these results show the competitiveness of the method proposed in this embodiment.
[0066] Example 2
[0067] Based on Example 1, the method of the present invention can also evaluate the existing timely defect localization method on the same data set, using five performance indicators from the field of timely defect localization research (i.e., Top-5accuracy, Top-10accuracy, Recall@20%Effort, Effort@20%Recall and Initial False Alarm) to automatically evaluate the quality of the model.
[0068] Table 2 Comparison of results of the method of the present invention and other methods
[0069]
[0070] Experiments show that the timely defect prediction method based on contrastive learning and feature fusion proposed in this embodiment can perform more accurate timely defect location compared with the baseline method. Specifically, the method of this embodiment combines expert features and semantic features extracted by CodeBERT, introduces contrastive learning in the semantic feature extraction stage, and constructs a contrast loss function. The expert features and semantic features are fused through feature fusion, so that the model can better identify the code change location, further improve the accuracy of timely defect location, and can surpass these baseline methods; for Top-5, the method of this embodiment can at least improve the performance by 8%; for Top-10, the method of this embodiment can at least improve the performance by 11.6%; for R@20%E, the method of this embodiment can at least improve the performance by 2.4%; for E@20%R, the method of this embodiment achieves the same performance as the current optimal method; for IFA, the method of this embodiment can at least improve the performance by 4.8%; these results show the competitiveness of the method proposed in this embodiment.
[0071] Example 3
[0072] Based on Example 1, the data set is repartitioned, and five performance indicators from the field of timely defect prediction research (i.e., F1-score, AUC, Recall@20% Effort, Effort@20% Recall, and POpt) are used to automatically evaluate the quality of the model.
[0073] Table 3 Comparison of results of the method of the present invention and other methods
[0074]
[0075] Experiments show that the timely defect prediction method based on contrastive learning and feature fusion proposed in this embodiment is higher than the baseline method in all indicators and can make more accurate timely defect prediction. Specifically, the method of this embodiment combines expert features and semantic features extracted by CodeBERT, introduces contrastive learning in the semantic feature extraction stage, and constructs a contrast loss function. The expert features and semantic features are fused through feature fusion, so that the model can better distinguish similar code changes, further improve the accuracy of timely defect prediction, and surpass these baseline methods; among them, for F1-score, the method of this embodiment can at least improve the performance by 15.8%; for AUC, the method of this embodiment can at least improve the performance by 5.5%; for R@20%E, the method of this embodiment can at least improve the performance by 11.9%; for E@20%R, the method of this embodiment can at least improve the performance by 16.7%; for Popt, the method of this embodiment can at least improve the performance by 3.9%; these results show the competitiveness of the method proposed in this embodiment.
[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A timely defect prediction method based on contrastive learning and feature fusion, characterized in that: The following steps are involved: 1.1: Preprocess and split the data set into training set D1 and test set D2; 1.2: Extract semantic features from the submission information and modification descriptions of the training set D1 divided in step 1.1; 1.3: Apply contrastive learning and define positive samples z p and negative samples z n , by maximizing the similarity between positive samples and minimizing the similarity between positive samples and negative samples, the model's ability to distinguish similar code changes is enhanced; 1.4: Calculate the positive sample z defined in step 1.3 p and z of negative samples n The contrast loss value of is obtained to obtain the total contrast loss, which is used to optimize the model; 1.5: In the feature fusion stage, the semantic features extracted in step 1.2 are fused with the expert features, and the model architecture is optimized by fine-tuning the fully connected layer and activation function; 1.6: Use the trained model to predict defects on the data in the test set D2 and output the prediction results.
2. The timely defect prediction method based on contrastive learning and feature fusion according to claim 1 is characterized in that: The step 1.2 comprises the following steps: 2.1: Use the CodeBERT model to convert each code snippet in the training set D1 into a feature vector. By inputting the context information of the code snippet, the semantic information in the code is captured and a feature vector of multiple dimensions is generated to represent the semantic content and structure of the code.
3. The timely defect prediction method based on contrastive learning and feature fusion according to claim 1 is characterized in that: The step 1.3 comprises the following steps: 3.1: Consider code changes with the same defect labels in training set D1 as positive samples z p , consider code changes with different defect labels as negative samples z n .
4. The timely defect prediction method based on contrastive learning and feature fusion according to claim 1, characterized in that: The step 1.4 comprises the following steps: 4.1: First, sample z i The embedding representation and positive sample z p The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z p ); 4.2: Then the sample z i The embedding representation and negative sample z n The embedding representation is calculated by dot product to obtain the similarity measure sim(z i ,z n ); 4.3: Bring the similarity measure of positive samples and negative samples into the contrast loss function, and finally calculate the total contrast loss L contrast , the formula is as follows: where z i represents the embedding of sample i, z p Indicates that z i The embedding of positive samples with the same label, z n Indicates that z i Embeddings of negative samples with different labels, sim represents the similarity measure between samples.
5. The timely defect prediction method based on contrastive learning and feature fusion according to claim 1 is characterized in that: The step 1.5 comprises the following steps: 5.1: Standardize the semantic features and expert features extracted from step 1.2 so that the two types of features can be fused at the same scale; 5.2: Concatenate the expanded semantic features and expert features to generate a comprehensive feature representation that integrates the information of semantic and expert features; 5.3: Input the concatenated comprehensive features into the multi-layer fully connected layer, and use the activation function to perform nonlinear transformation on the features; 5.4: Based on the fully connected layer, the back propagation algorithm is used to update the weights of the model and adjust the parameters of the fully connected layer and activation function to maximize the predictive performance of the model.
6. The timely defect prediction method based on contrastive learning and feature fusion according to claim 1, characterized in that: In step 1.6, the trained model is used to predict defects on the data in the test set D2 and the prediction results are output.