Patch overfitting evaluation method combined with dual code representation

Abstract

The invention relates to a patch overfitting evaluation method combined with dual code representation, and belongs to the field of internet security services. Aiming at an automatic patch generation condition, firstly, a defect code snippet, a correct code snippet and CFG and AST of a patch are constructed, and a Word2Vec model is utilized to respectively generate embedded vectors containing structural information and semantic information; secondly, a Transform model is utilized to encode an AST embedded vector, a GAT model is utilized to encode a CFG embedded vector, and a feature vector is obtained through splicing; then calculating feature vector distances between the patches and the defect code snippets and between the patches and the correct code snippets, and training to generate a dichotomy model for calculating patch overfitting confidence; and finally, weighting the confidence coefficient according to the code change degree, and comparing the weighted confidence coefficient with a threshold value to judge whether overfitting exists. Aiming at the defects in the existing method, the semantic similarity of correct code snippets is high, and misjudgment is easily caused when the code change degree is large, the code CFG features are increased, the code change degree is calculated, and the patch overfitting evaluation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to a method for evaluating patch overfitting by combining dual code representations, belonging to the technical field of computer programs. Background Art

[0002] Automated Program Repair (APR) technology can repair software defects without manual intervention, alleviating the workload of technicians and being of great significance to software development and maintenance. However, the patches generated by APR tools may have overfitting problems, that is, although the patches pass the test cases, the defect vulnerabilities are not completely solved. Currently, to evaluate the correctness of such patches, researchers have proposed various methods such as static analysis, dynamic analysis, and deep learning-based methods. Static methods rely on the static features of the code, such as code change patterns, syntactic or semantic similarities, and can be evaluated without executing tests. Such methods have high requirements for the capabilities of technicians and require high time costs, so they are rarely used in practical applications. Dynamic methods need to execute test cases to evaluate the correctness of the patches. The method evaluates by checking whether the additional test cases generated by automated test case generation tools (such as Randoop, Evosuite, etc.) pass, or by collecting program runtime information. Such methods have specific requirements for test cases and need to detect the program running and collect information, resulting in long running times and low efficiency, so they are also rarely used in practical applications.

[0003] Deep learning-based methods convert source code into modalities such as token sequences or Abstract Syntax Trees (ASTs), and perform embeddings. By calculating the similarities between the code with the patch to be evaluated and the correct code and defective code provided by developers, the correctness of the patch is judged. Such methods can capture semantic or syntactic information of the code and are trained using a large amount of code data, thereby improving the accuracy of the model in evaluating the correctness of patches. Such methods have short running times and high running efficiencies and are widely used.

[0004] However, the deep learning-based patch correctness evaluation method has the following problems: (1) Using a single modality (such as token sequences, ASTs, etc.) for feature extraction results in the embedding vectors not being able to well capture the code features, causing the vector representations of defective code fragments and correct code fragments in the feature space to have weak differences and being difficult to distinguish. For example, using only AST for feature extraction involves a large number of fine-grained syntax nodes, and it is challenging to identify and capture semantically meaningful features; (2) Only using code similarity for evaluation ignores the correlation between patch correctness and code change degree, resulting in a low recall rate of the evaluation method. Therefore, the present invention proposes a method for evaluating patch overfitting by combining dual code representations. Summary of the Invention

[0005] In view of the high semantic similarity between defective code fragments and correct code fragments in existing methods and the easy misjudgment when the degree of code change is large, the present invention proposes a patch overfitting evaluation method combining dual code representations.

[0006] The design principle of the present invention is as follows: First, construct the abstract syntax tree and control flow graph (Control Flow Graph, CFG) of the triple (patched code fragment, defective code fragment, correct code fragment), and use the Word2Vec model to generate embedding vectors containing structural information and semantic information respectively; secondly, use the encoder based on Transformer to encode the AST embedding vector, and the encoder based on Graph Attention Networks (GAT) to encode the CFG embedding vector, and splice them to obtain a feature vector; then calculate the distance vectors between the patched code fragment (the patch is generated by the APR tool for the target defective code fragment) and the correct code fragment (provided by the developer for the defective code fragment), and between the defective code fragments through a comparison function, and train to generate a binary classification model for calculating the patch overfitting confidence; finally, weight the confidence according to the code change degree and compare it with the threshold to judge whether there is overfitting.

[0007] The technical solution of the present invention is realized through the following steps:

[0008] Step 1, for the patched code fragment P, the correct code fragment C, and the defective code fragment B, generate the AST and CFG respectively, and construct the feature vector V AST and the feature vector V CFG .

[0009] Step 1.1, traverse the AST by the depth-first algorithm, record the nodes encountered during the traversal in order to form a text sequence, then establish a mapping for each element in the sequence, and then train the text vector through the Word2Vec model to generate the AST embedding vector, and finally use the encoder based on Transformer to obtain the feature vector V AST .

[0010] Step 1.2, for the CFG, use Word2Vec and Bi-LSTM to capture the sequence dependencies in the statement words, generate an initial embedding vector for each CFG node, capture the relationships between nodes through GAT, and process the output of the GAT layer through a feed-forward neural network and an Add&Norm layer to obtain the feature vector V CFG .

[0011] Step 2, splice the feature vectors of the two code representations to obtain V code, then calculate the distance vector D.

[0012] Step 2.1, calculate the difference between P and B, quantify the difference through comparison functions such as element-wise squared difference and element-wise product, and splice to obtain the distance vector D(Patch, Buggy).

[0013] Step 2.2, calculate the distance vector D(Patch, Correct) between P and C in the same way as above.

[0014] Step 3, use a logistic regression model with the sigmoid function as the activation function, and predict the probability of patch overfitting according to the two distance vectors to obtain the patch overfitting confidence.

[0015] Step 4, from the perspectives of code change degree and code similarity, determine whether to weight the penalty factor for the patch overfitting confidence, and compare with the threshold to obtain the evaluation result.

[0016] Step 4.1, calculate the code similarity s between P and C. If the similarity is lower than the similarity threshold t 1 , then weight the penalty factor λ for the overfitting confidence 1 .

[0017] Step 4.2.1, calculate the code change degree d between B and P, C patch,bug , d correct,bug .

[0018] Step 4.2.2, given the initial change degree threshold d 0 , and then determine the final change degree threshold t according to d correct,bug . If d 2 exceeds the threshold, then weight the penalty factor λ for the overfitting confidence patch,bug . 2 .

[0019] Step 4.3, obtain the final overfitting confidence from the above steps, compare with the overfitting threshold T, and output the prediction result.

[0020] Beneficial effects

[0021] Compared with other patch correctness evaluation methods that use single-modal for feature extraction, the present invention uses an encoder based on Transformer to process the abstract syntax tree and an encoder based on graph attention network to process the control flow graph, and implements the multi-head attention mechanism in both double encoders, enabling the model to effectively capture features such as semantics, syntax, and control flow of the code at both the token level and the statement level, thereby providing highly reliable prediction results during overfitting evaluation.

[0022] Compared with other patch correctness evaluation methods that only consider the similarity of code vector representations, the present invention introduces two penalty factors based on code change degree and code similarity, and improves the accuracy of code change measurement by combining Euclidean distance, edit distance, and AST syntax node differences, thereby enhancing the effectiveness of the evaluation method in the face of different code change scenarios. Description of the Drawings

[0023] Figure 1 It is a flowchart of the patch overfitting evaluation method combining dual code representations of the present invention. Detailed Implementation Manner

[0024] To better illustrate the purpose and advantages of the present invention, the implementation manner of the method of the present invention will be further described in detail below with reference to examples.

[0025] The experimental data mainly comes from the labeled patches generated by existing open-source APR tools for defects in Defects4J and Defects4J.

[0026] Defects4J is a database containing real-world Java program defects. It provides bug sets of multiple well-known open-source Java projects, and each bug has corresponding test cases to show the behavioral differences before and after repair.

[0027] Table 1 Information of Evaluation and Training Datasets

[0028]

[0029] The experiment uses recall and precision to evaluate the detection performance of the present invention for patch overfitting. Among them, recall represents the proportion of overfitting patches correctly identified among all overfitting patches; precision represents the proportion of overfitting patches correctly identified among all patches identified as overfitting patches. The calculation methods of recall Recall and precision Precison are as shown in the formula:

[0030]

[0031] Among them, TP is the number of overfitting samples predicted as overfitting samples, FN is the number of overfitting samples predicted as correct samples, FP is the number of correct samples predicted as overfitting samples, and TN is the number of correct samples predicted as correct samples.

[0032] The specific process is as follows:

[0033] Step 1, for each patched code snippet P, correct code snippet C, and defective code snippet B in the training set, construct their ASTs and CFGs respectively.

[0034] Step 1.1, taking P as an example, first serialize the AST through DFS (Depth-First Search) to obtain a sequence X = {x 1 , x 2 , …, x t}, where xi represents the i-th AST node, t is the maximum sequence length. For sequences with lengths less than t, pad them with zeros and mask them during processing, record the type and attributes of each node, and obtain a text sequence vector. Secondly, establish a mapping, assign a unique integer to each different symbol that appears during the serialization process to achieve node tokenization and obtain a digital vector. Then use the Word2Vec model to train the text vector to generate an AST embedding vector and obtain an initial embedding vector matrix Apply positional encoding to inject relative order information into the AST vector. Finally, input the processed into the multi-head attention layer and point-wise feed-forward layer of the Transformer, and there is an Add&Norm sub-layer behind each layer, and the output is as follows:

[0035] where represents the node embedding vector of x i obtained after passing through the multi-head attention module.

[0036] Step 1.2, also taking P as an example, first convert the nodes in the CFG into a node set N = {n 1 , n 2 , …, n k}, and obtain an adjacency matrix M by converting the CFG into a directed graph with single-type edges, where n i represents the i-th CFG node. Secondly, segment each code statement (node of the CFG) into a token sequence, and use Word2Vec to convert the token sequence into a fixed-size vector representation. Then input the vector generated by Word2Vec into a Bi-LSTM to generate an initial embedding vector matrix Input and M into an encoder based on GAT, which includes a GAT, a feed-forward layer, and an Add&Norm layer, and the output is as follows:

[0037]

[0038] where represents the vector corresponding to the n i node.

[0039] Step 1.3, calculate the feature vectors corresponding to B and C in the same way as above.

[0040] Step 2, concatenate the feature vectors of the two code representation methods to obtain Calculate the distance vectors D(Patch, Correct) and D(Patch, Buggy) respectively.

[0041] Step 2.1.1: Taking the distance vector D(Patch, Buggy) as an example, it consists of three parts: the square of the element-wise difference, the element-wise product, and the Euclidean distance.

[0042] Step 2.1.2: Calculate the square of the element-wise difference. The formula is as follows:

[0043]

[0044] , where ⊙ represents element-wise multiplication (i.e., Hadamard product).

[0045] Step 2.1.3: Calculate the element-wise product. The formula is as follows:

[0046]

[0047] Step 2.1.4: Calculate the Euclidean distance. The formula is as follows:

[0048]

[0049] , where ||*|| represents the Frobenius norm.

[0050] Step 2.1.5: The distance vector is defined as follows:

[0051]

[0052] , where ⊕ represents the concatenation operation.

[0053] Step 2.2: Calculate the distance vector D(Patch, Correct) between P and C in the same way as D(Patch, Buggy).

[0054] Step 3: Use the distance vectors D(Patch, Buggy) and D(Patch, Correct) as the inputs of the prediction model using the logistic regression algorithm to obtain the overfitting confidence p of the patch to be evaluated. Initial .

[0055] Step 4: Apply a weighted penalty factor to the overfitting confidence from the code similarity and code change degree, and compare it with the overfitting threshold to obtain the prediction result.

[0056] Step 4.1: Calculate the code similarity between P and C. The relevant formula is as follows:

[0057]

[0058] t 1Let \(\theta\) be the similarity threshold, which needs to be adjusted according to the experimental results. Selecting appropriate similarity threshold and penalty coefficient \(\lambda_1\) can help improve the accuracy of the evaluation method.

[0059] Step 4.2: Calculate the code change degree \(d\) patch,bug and \(d\) correct,bug , and dynamically determine the current change degree threshold \(t\) according to the latter 2 , and the relevant formula is as follows:

[0060] \(t\) 2 = \(k \cdot d\) avg (9)

[0061]

[0062] where \(d\) avg is the weighted average of the change degrees between all defective code snippets and the corresponding correct code snippets in the dataset, \(k\) is the coefficient controlling the change degree threshold, usually a small value; \(min(*)\) is the minimum value of the corresponding item in the dataset, \(max(*)\) is the maximum value of the corresponding item in the dataset; is the Levenshtein distance (i.e., edit distance) between \(C\) and \(B\), is the difference in the number of AST nodes, is the Euclidean distance between \(C\) and \(B\); \(w\) 1 , \(w\) 2 , \(w\) 3 are weight factors, and the sum of the three is 1. (Note: The edit distance is a method to measure the difference between two strings, which is defined as the minimum number of single-character edit operations required to convert one string into another string. In the present invention, the AST sequence obtained in Step 1.1 can be used and calculated by using the dynamic programming algorithm or directly calling the corresponding library function.)

[0063] Step 4.3: Obtain the final overfitting confidence from the above steps:

[0064] \(p = p\) Initial * \(\lambda\) 1 * \(\lambda\) 2 (15)

[0065]

[0066] where \(T\) is the overfitting threshold.

[0067] Table 2 shows the comparison of the effects between the present invention and other existing APCA methods based on deep learning or machine learning algorithms. It can be found from Table 2 that, compared with the BERT+LR method, the recall rate of the method of the present invention has increased by 0.36, and the precision rate has decreased by 0.03; compared with the ODS method, the recall rate of the method of the present invention has increased by 0.11, and the precision rate has increased by 0.04; compared with the INVALIDATORsyn method, the recall rate of the method of the present invention has increased by 0.07, and the precision rate has increased by 0.01.

[0068] Table 2 Experimental Results

[0069]

[0070] In summary, compared with the existing methods, the present invention can significantly improve the recall rate while ensuring a high classification precision rate, and can more comprehensively and accurately measure the repair effects of patches generated by different APR tools, which plays a guiding role in the improvement and optimization of related tools.

[0071] The above specific description further details the purpose, technical solution and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A patch overfitting evaluation method combined with dual code representation, characterized by The method comprises the following steps: Step 1: For the patched code snippet P, the correct code snippet C, and the defective code snippet B, construct AST and CFG respectively; for AST, establish a mapping for the traversed text sequence, generate an embedding vector through the Word2Vec model, and then use the Transformer-based encoder to encode the embedding vector to obtain V AST ; For CFG, Word2Vec and Bi-LSTM are used to generate initial embedding vectors for nodes, and then the embedding vectors are encoded using an encoder based on a graph attention network to obtain V CFG ; Splicing V AST and V CFG Get the feature vector V code ; Step 2: Calculate the distance vectors D(Patch, Correct) and D(Patch, Buggy) between P and C and B, and use comparison functions to capture the differences between them and concatenate them to obtain the distance vector. The comparison functions include element-by-element square difference subtraction, element-by-element multiplication, and Euclidean distance. Step 3, using the distance vector as input to a machine learning model using a logistic regression algorithm to obtain the patch overfitting confidence; Step 4: According to the code change degree and code similarity, determine whether to add a weighted penalty factor to the patch overfitting confidence, and compare it with the threshold to obtain the evaluation result.

2. The patch overfitting evaluation method combined with dual code representation according to claim 1, characterized in that: In step 1, the feature extraction stage jointly represents the code through AST and CFG, and for AST and CFG, a Transformer-based encoder and a graph attention network-based encoder are used for processing respectively.

3. The patch overfitting evaluation method combined with dual code representation according to claim 1, characterized in that: In step 4, the penalty factor is introduced from the perspective of code change degree and code similarity, and the formula for calculating code change degree is t2=k*d avg (1) Among them, d avg is the weighted average of the degree of change between all defective code snippets and the corresponding correct code snippets in the data set; k is the coefficient for controlling the threshold of the degree of change, which is usually small; min(*) is the minimum value of the corresponding item in the data set, and max(*) is the maximum value of the corresponding item in the data set; is the Levenshtein distance (i.e., edit distance) between the correct code fragment C and the defective code fragment B; is the difference in the number of AST nodes between the correct code fragment C and the defective code fragment B, is the Euclidean distance between the correct code fragment C and the defective code fragment B, w1, w2, and w3 are weight factors, and their sum is 1.

Images