Graph neural network program defect prediction method based on counterexample training
By using a graph neural network method based on negative examples for training, the problem of insufficient accuracy of graph neural networks in program defect prediction is solved. By generating negative examples for model training, the model's ability to learn program semantics and its prediction accuracy are improved, thus achieving more efficient program defect detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2023-12-13
- Publication Date
- 2026-07-21
Smart Images
Figure CN117669658B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of software engineering, programming languages and deep learning, and involves methods for learning program semantics and predicting program properties. Specifically, it relates to a method for predicting program defects based on graph neural networks trained with counterexamples. Background Technology
[0002] With the rapid development of computer technology, the number of software programs is increasing dramatically, and open-source OSS is being widely used. Consequently, the number and impact of software defects are also increasing and expanding. Software defects pose a potential threat to the secure operation of computer systems, can cause huge economic losses to society, and may also lead to serious information leakage problems. Therefore, further research into program defect detection technology is needed.
[0003] While defect detection techniques based on static analysis, dynamic testing, and formal methods have undergone decades of development, they still suffer from low efficiency and high costs. Some work has leveraged machine learning techniques to reduce the cost of defect detection or to predict defects. Currently, with the development of deep learning technology, Graph Neural Networks (GNNs), as a type of neural network model, are widely used in fields such as social networks, recommender systems, financial risk control, physical systems, molecular chemistry, life sciences, knowledge graphs, and traffic prediction. Simultaneously, in research on program defect detection, scholars have shown increasing interest and attention to the performance of GNNs and related models. GNNs have achieved remarkable results in program analysis, such as variable misuse prediction, function name prediction, and defect detection and repair, where their performance outperforms other text- or sequence-based methods.
[0004] Predicting program defects using deep learning technology has gradually become a research direction in academia. However, the data representation of existing program code lacks program structure information, and the complex program structures such as numerous branches, loops, recursions, and function calls increase the difficulty for deep learning technology to understand program semantics, resulting in the insufficient ability of existing models to predict program defects.
[0005] Early work primarily treated program source code as text and used natural language processing techniques to find superficial, text-based patterns within the program. Subsequent methods aimed to learn semantically informational program embeddings from Abstract Syntax Trees (ASTs). Despite these advancements, these tree-based models remained limited to learning the syntactic features of programs due to the limited information provided by ASTs. Recently, another model called Graph Neural Networks (GNNs) has been introduced into the programming field. Its main idea is to simplify the problem by learning from the graph representation of the program, rather than directly learning the program representation from the source code. With more efficient message passing methods, GNNs have achieved results in various domains, such as variable misuse prediction, function name prediction, and error detection and correction. Although GNNs have significantly improved upon previous work, their accuracy and efficiency remain to be addressed, with the following main challenges:
[0006] The first challenge is that the accuracy of Generative Neural Networks (GNNs) in predicting program defects remains low, a problem that urgently needs to be addressed. Program defects pose a significant potential threat to software system security, potentially leading to financial losses and security risks. Therefore, efficient and accurate prediction of program defects is crucial for ensuring software system security. However, while current research demonstrates the feasibility of applying GNNs in program defect prediction, precision and recall remain unsatisfactory, with many defects being missed or falsely detected. Therefore, improving the accuracy of GNNs in program defect prediction is of paramount importance.
[0007] The second challenge is that GNNs may miss some unique structural features in the graph representation of a program during the basic training process. When certain features that can characterize program defects are not learned by the GNN, the prediction accuracy of the GNN on the validation set for samples with these features will be affected. Therefore, it is necessary to strengthen the GNN to learn the features missed during the training process.
[0008] The third challenge is that GNNs struggle to learn from graphs with a large number of nodes due to the high cost of their basic message passing process. Ideally, each node should pass messages directly or indirectly to every other node in the graph to achieve sufficient information exchange. However, this expensive information propagation process is difficult to scale to graphs with many nodes without loss of accuracy. This is because while message passing technology gives GNNs powerful learning capabilities, it hinders their ability to learn the information contained in nodes that are far apart in the graph. From the perspective of information propagation, a message sent from the starting node needs to pass through many intermediate nodes to reach the ending node. Due to aggregation operations, the message is diluted as soon as it is absorbed by intermediate nodes to update their own states. By the time the message reaches the ending node, its accuracy will be severely reduced. In the domain of programming languages, long-distance dependencies are very common due to the complexity of code functionality and logic. This also makes it more difficult for neural network models to learn from the graph representation of programs. Therefore, to improve learning performance, it is necessary to be able to learn the features of this relationship. To solve the long-distance dependency problem, existing work has designed additional edges in the Abstract Syntax Tree (AST) to connect nodes with certain relationships (e.g., data or control dependencies). However, this work only discusses the effectiveness of its method in theory and does not prove the scalability of the model through rigorous testing; it also does not give specific instructions on how to add dependency edges to make the model have optimal scalability. Summary of the Invention
[0009] To address the problems in existing technologies, this invention proposes a graph neural network program property prediction method based on counterexample training. This method enables the graph neural network to enhance its learning of program graph features not learned during training, thereby improving the accuracy and capability of program property prediction.
[0010] To solve the above problems, the present invention adopts the following technical solution:
[0011] A graph neural network-based method for predicting program defects based on counterexample training is proposed. This method targets a program property prediction task. First, it collects the original dataset related to program properties, converts the program source code into a graph abstract representation to obtain an experimental dataset, and then trains a graph neural network model based on the experimental dataset. Subsequently, based on samples that the graph neural network model cannot correctly predict in the training dataset, it performs random mutations to generate similar counterexample samples, which are added to the training set of the graph neural network model to further train the model. Finally, the trained model is used to predict program properties.
[0012] The above-mentioned graph neural network program defect prediction method based on counterexample training is characterized by the following steps:
[0013] Step 1) Define a program property prediction task and collect the original dataset corresponding to the program property prediction task to train a method for predicting the existence of a specific program property; the content of each data sample includes the source code of the program method (function) and the smallest source code fragment that can characterize the specific program property;
[0014] Step 2) Convert the program source code in the original dataset into a graph abstract representation to obtain the experimental dataset. By constructing a control flow graph and an abstract syntax tree of the corresponding statements, the program source code is transformed into a graph representation with control flow edges and abstract syntax edges to obtain the experimental dataset.
[0015] Step 3) Divide the obtained experimental dataset into a training set and a validation set, and use the training set as input to train the graph neural network model, and use the validation set to evaluate the model's performance; the output of the model is whether a specific sample has a specific program property;
[0016] Step 4) Using the prediction results of the trained graph neural network model on the training set samples, select the sample data with incorrect predictions to generate counterexample samples; generate similar data samples by randomly mutating the source code corresponding to the sample with incorrect predictions, give them real program property labels, and then perform static analysis to confirm them. Add the confirmed samples to the training set of the model and carry out the next round of graph neural network model training. Repeat the feedback training process multiple times to improve the generalization ability and prediction effect of the model.
[0017] Step 5) Predict the program properties in the actual program using the trained model; first calculate the state vector of each node, then calculate the final representation of the entire graph by summarizing the states of all the nodes; finally, train the feedforward neural network and the representation prediction method of the entire graph to determine if there are specific properties.
[0018] The above-mentioned graph neural network program defect prediction method based on counterexamples is further characterized in that step 2) specifically includes the following steps:
[0019] Step 21) Construct a control flow graph; Given a function, obtain the control flow graph of functions involved in specific program characteristics by analyzing the program source code, and then split each basic block into multiple nodes, each node being a statement within the basic block. This method allows for the analysis of specific statements; When analyzing a function, concatenate all calling functions and called functions of the function into the graph of the function to form a complete function call chain, which is used to describe the context relationship of the function.
[0020] Step 22) Constructing the abstract syntax tree. This embodiment first obtains the statement corresponding to the node, then constructs the abstract syntax tree for the corresponding statement. Each node in the abstract syntax tree is converted into an index (represented by a positive integer) using a predefined dictionary. For each node v...i The node v i Represented as (t1, t2, ..., t j ), where j is the number of nodes, t m m∈{1,...,j} is the index of the node in the dictionary; finally, the feature value of the node is determined by a hash function.
[0021] Step 23) Form a graph abstract representation of the source code. There are two types of edges between nodes: control flow edges between nodes are obtained when analyzing the control flow graph of the functions involved; and abstract syntax edges between nodes are obtained when constructing the abstract syntax tree of the statements corresponding to the nodes. These two types of edges constitute the edges between nodes in the program graph abstract representation.
[0022] Step 4) specifically includes the following steps:
[0023] Step 41) Based on the initial training results of the GGNN model, select training set samples that the trained model cannot correctly predict program properties to generate counterexamples;
[0024] Step 42) Based on the samples selected in Step 41), determine the key statements through the corresponding minimum source code fragments in Step 1). All program statements outside the program source code fragments are considered irrelevant statements. The definition of a key statement is: the set of minimum program statements in the sample program that can represent a specific program property, which is the minimum source code fragment in the original dataset in Step 1) that can represent a specific program property. The definition of an irrelevant statement is: the set of program statements in the sample program that will not affect the specific program property, which is the set of statements in the source code of each sample in the original dataset in Step 1) after removing all key statements.
[0025] Step 43) Select a sample from Step 41), and use the set of key statements in this sample as the initialization negative sample. Define the initialization negative sample as the round 0 negative sample; obtain the number of irrelevant statements in this sample as N; number the irrelevant statements as irrelevant statement 1, irrelevant statement 2, ..., irrelevant statement n.
[0026] Step 44) In the m-th training round, select an irrelevant statement m and perform random mutation on it. m is initialized to 1 and then incremented to n. Mutation methods include adding other statements before or after the statement, modifying variable names, types, or values, and modifying judgment conditions. The resulting set of mutated statements is then added to the samples of the (m-1)-th round to generate the counterexample samples for the m-th round.
[0027] Step 45) Perform static analysis on the counterexamples obtained in Step 44) to confirm that the generated counterexamples have the specified program properties. If the generated counterexamples do not have the specified program properties, return to Step 44) to regenerate counterexamples based on the irrelevant statement m; if the generated counterexamples have the specified program properties, proceed to the next step.
[0028] Step 46) After converting the counterexample sample into a graph abstract representation through the program, add it to the training set of the graph neural network model for training. Determine whether the model can correctly predict the specific program properties of the original sample corresponding to the counterexample sample. If the graph neural network model can correctly predict the original sample, terminate the counterexample training of the original sample; otherwise, proceed to step 44) to continue the counterexample training.
[0029] The above-mentioned graph neural network program defect prediction method based on counterexamples is characterized in that: the graph neural network model adopts a gated graph neural network, namely the GGNN model, whose input is the graph abstract representation after the program source code is converted. The conversion method is to first construct the statement-level control flow graph of the program, at which point each basic block is a statement, and then represent each statement through an abstract syntax tree.
[0030] The above-mentioned graph neural network program defect prediction method based on counterexamples is characterized in that: the counterexample sample is defined as: a new data sample generated by randomly mutating and modifying statements based on samples that the graph neural network model cannot correctly predict in the training dataset.
[0031] The present invention adopts the above technical solution and has the following beneficial effects:
[0032] This invention enhances the ability of graph neural network models to learn program semantics and predict program defects, thereby improving the prediction performance of program property prediction tasks such as correctness and security. The invention's method of converting program source code into a graph abstraction strengthens the graph neural network's ability to learn program semantics, thus improving its learning capacity. Furthermore, by using counterexamples to further train the graph neural network, it enhances the learning of program graph features not learned during training, thereby improving the accuracy and capability in program property prediction. Attached Figure Description
[0033] Figure 1 This is a framework diagram of the graph neural network program defect prediction method based on counterexample training according to an embodiment of the present invention.
[0034] Figure 2 This is a specific implementation framework diagram of the graph neural network model based on the iterative training model using counterexamples, according to an embodiment of the present invention. Detailed Implementation
[0035] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0036] This embodiment provides a graph neural network-based method for predicting program defects based on counterexample training. Figure 1 The main framework and overview of each module of this invention are illustrated: 1) The input to the model is the program source code and the smallest source code fragment information that can characterize a specific program property. 2) Program conversion: The program conversion module analyzes the input (program source code and fragments), constructs a control flow graph, and then converts the input into a graph abstract representation based on the program's control flow graph and the abstract syntax tree of each program statement. Finally, the graph abstract representation of the program under test is passed to the GGNN model. 3) GGNN model: A graph neural network model trained based on the graph abstract representation information of the program under test. 4) Counterexample generation: This module selects data samples that the GGNN model cannot correctly predict specific program properties on the training set for counterexample generation. The data samples that cannot be correctly predicted are used as the original samples for counterexample mutation. Through random mutation, an irrelevant statement in the original sample is modified to generate a counterexample sample, which is then added to the training set to continue training the GGNN model. 5) Program property prediction: The program property prediction module passes the graph abstract representation of the program source code and corresponding program property information to the GGNN model to query the model's prediction results and finally outputs the model's prediction results.
[0037] The graph neural network program defect prediction method based on counterexample training in this embodiment specifically includes the following steps:
[0038] Step 1) Creating the Original Dataset. Define a program property prediction task and collect the corresponding original dataset. The original dataset contains not only the source code of methods (functions) with specific program properties, but also the source code of methods (functions) without such program properties. The ratio of samples with and without specific program properties in the original dataset should be approximately 3:1.
[0039] Step 2) Program Transformation. Given a program, the corresponding control flow graph information is generated using program analysis techniques. The input to the GGNN model is in the form of a graph. Although the control flow graph is a graph representation of a program, to obtain more accurate program semantic information, it is necessary to design more specific representations for each basic block, each statement, and even each expression. Therefore, this method proposes a new program representation method based on program analysis techniques, as detailed below:
[0040] The graph abstraction of a program: Given a function, the control flow graph of functions related to specific program characteristics is obtained by analyzing the program source code. Then, each basic block is broken down into multiple nodes, each node being a statement within the basic block. This allows for the analysis of specific statements. When analyzing a function, all calling and called functions are concatenated into the function's graph to form a complete function call chain, used to describe the function's context. This embodiment first obtains the statements corresponding to the nodes, then constructs an abstract syntax tree for the corresponding statements. Each node in the abstract syntax tree is converted into an index (represented by positive integers) using a predefined dictionary. For each node v... i The node v i Represented as (t1, t2, ..., t j ), where j is the number of nodes, t m m∈{1,...,j} is the index of the node in the dictionary; finally, a hash function is used to determine the feature value of the node. There are two types of edges between nodes: control flow edges between nodes are obtained when analyzing the control flow graph of the functions involved; abstract syntax edges between nodes are obtained when constructing the abstract syntax tree of the statements corresponding to the nodes. These two types of edges constitute the edges between nodes in the abstract representation of the program graph.
[0041] Finally, the abstract representation of the program graph obtained from the program transformation is passed to the GGNN model for training and prediction, which transforms the original dataset into an experimental dataset.
[0042] Step 3) Train the GGNN model. Divide the experimental dataset obtained in Step 2) into a training set and a validation set. Use the training set as input to train the GGNN model, and use the validation set to evaluate the model's performance. The model's output indicates whether a specific sample possesses a specific procedural property.
[0043] Step 4) Counterexample Generation. Using the prediction results of the trained graph neural network model on the training set samples, select the sample data with incorrect predictions to generate counterexample samples; generate similar data samples by randomly mutating the source code corresponding to the samples with incorrect predictions, give them real program property labels, and then perform static analysis to confirm them. Add the confirmed samples to the model's training set for the next round of graph neural network model training. Repeat the feedback training process multiple times to improve the model's generalization ability and prediction effect.
[0044] Step 5) Program Property Prediction. Predict the program properties in the actual program using the trained model; first, calculate the state vector of each node, then calculate the final representation of the entire graph by summarizing the states of all the nodes; finally, train the feedforward neural network and the representation prediction method of the entire graph to determine if there are specific properties.
[0045] The aforementioned graph neural network-based method for predicting program defects based on counterexamples, Figure 2 This demonstrates the specific process and approach for generating counterexamples in step 4). It includes the following steps:
[0046] Step 41) Based on the initial training results of the GGNN model, select training set samples that the trained model cannot correctly predict program properties to generate counterexamples;
[0047] Step 42) Based on the samples selected in Step 41), determine the key statements through the corresponding minimum source code fragments in Step 1). All program statements outside the program source code fragments are considered irrelevant statements. The definition of a key statement is: the set of minimum program statements in the sample program that can represent a specific program property, which is the minimum source code fragment in the original dataset in Step 1) that can represent a specific program property. The definition of an irrelevant statement is: the set of program statements in the sample program that will not affect the specific program property, which is the set of statements in the source code of each sample in the original dataset in Step 1) after removing all key statements.
[0048] Step 43) Select a sample from Step 41), and use the set of key statements in this sample as the initialization negative sample. Define the initialization negative sample as the round 0 negative sample; obtain the number of irrelevant statements in this sample as N; number the irrelevant statements as irrelevant statement 1, irrelevant statement 2, ..., irrelevant statement n.
[0049] Step 44) In the m-th training round, select an irrelevant statement m and perform random mutation on it. m is initialized to 1 and then incremented to n. Mutation methods include adding other statements before or after the statement, modifying variable names, types, or values, and modifying judgment conditions. The resulting set of mutated statements is added to the samples of the (m-1)-th round to generate the counterexample samples for the m-th round.
[0050] Step 45) Perform static analysis on the counterexamples obtained in Step 44) to confirm that the generated counterexamples possess the specified program properties. If the generated counterexamples do not possess the specified program properties, return to Step 44) to regenerate counterexamples based on the irrelevant statement m. If the generated counterexamples possess the specified program properties, proceed to the next step.
[0051] Step 46) Add the counterexample sample to the training set of the graph neural network model for training. Determine whether the model can correctly predict the specific program properties of the original sample corresponding to the counterexample sample. If the graph neural network model can correctly predict the original sample, terminate the counterexample training of the original sample; otherwise, proceed to step 44) to continue the counterexample training.
[0052] The above descriptions are embodiments of the present invention, but the specific embodiments described herein are merely illustrative and not intended to limit the invention. Any omissions, modifications, or equivalent substitutions made within the scope of the claims of this invention without departing from the principles and spirit of the invention should be included within the protection scope of this disclosure.
Claims
1. A method for predicting program defects based on graph neural networks trained with counterexamples, characterized in that, Specifically, the following steps are included: Step 1) Define a program property prediction task and collect the original dataset corresponding to the program property prediction task to train a method for predicting the existence of a specific program property; the contents of each original data sample include the source code of the program method and the smallest source code fragment that can characterize the specific program property; Step 2) Convert the program source code in the original dataset into a graph abstract representation. By constructing a control flow graph and an abstract syntax tree of the corresponding statements, the program source code is transformed into a graph representation with control flow edges and abstract syntax edges, thus obtaining the experimental dataset. Step 3) Divide the obtained experimental dataset into a training set and a validation set, and use the training set as input to train the graph neural network model, and use the validation set to evaluate the model's performance; the output of the graph neural network model is whether the sample has a specific program property; Step 4) Using the prediction results of the trained graph neural network model on the training set samples, select the sample data with incorrect predictions to generate counterexamples; generate similar data samples by randomly mutating the source code corresponding to the samples with incorrect predictions, assign them true program property labels, and then perform static analysis to confirm them. Add the confirmed samples to the model's training set for the next round of graph neural network model training. Repeat the feedback training process multiple times to improve the model's generalization ability and prediction effect; Step 4) specifically includes the following steps: Step 41) Based on the initial training results of the GGNN model, select training set samples from which the trained model cannot correctly predict program properties to generate counterexamples; Step 42) Based on the samples selected in Step 41), determine the key statements through the corresponding minimum source code fragments in Step 1). All program statements outside the program source code fragments are considered irrelevant statements. The definition of a key statement is: the set of minimum program statements in the sample program that can represent a specific program property, which is the minimum source code fragment in the original dataset in Step 1) that can represent a specific program property. The definition of an irrelevant statement is: the set of program statements in the sample program that will not affect the specific program property, which is the set of statements in the source code of each sample in the original dataset in Step 1) after removing all key statements. Step 43) Select a sample from Step 41), and use the set of key statements in this sample as the initialization negative sample. Define the initialization negative sample as the round 0 negative sample; obtain the number of irrelevant statements in this sample as N; number the irrelevant statements as irrelevant statement 1, irrelevant statement 2, ..., irrelevant statement n. Step 44) In the m-th training round, select an irrelevant statement m and perform random mutation on the statement; where m is initialized to 1 and then incremented to n; the mutation methods include adding other statements before and after the statement, modifying the variable name, variable type or variable value in the statement, and modifying the judgment condition; add the set of mutated statements to the samples of the (m-1)-th round to generate the counterexample samples of the m-th round; Step 45) Perform static analysis on the counterexamples obtained in Step 44) to confirm that the generated counterexamples have the specified specific program properties. If the generated counterexamples do not have the specific program properties, return to Step 44) to regenerate counterexamples based on the unrelated statement m. If the generated counterexamples have the specific program properties, proceed to the next step. Step 46) After converting the counterexample sample into a graph abstract representation through the program, add it to the training set of the graph neural network model for training. Determine whether the model can correctly predict the specific program properties of the original sample corresponding to the counterexample sample. If the graph neural network model can correctly predict the original sample, terminate the counterexample training of the original sample; otherwise, proceed to step 44) to continue the counterexample training.
2. The graph neural network program defect prediction method based on counterexample training according to claim 1, characterized in that, Step 2) specifically includes the following steps: Step 21) Constructing a control flow graph: Given a function, the control flow graph of functions involved in specific program characteristics is obtained by analyzing the program source code. Then, each basic block is split into multiple nodes, and each node is a statement within the basic block. In this way, specific statements can be analyzed. When analyzing a function, all calling functions and called functions of the function are concatenated into the graph of the function to form a complete function call chain, which is used to describe the context relationship of the function. Step 22) Constructing the Abstract Syntax Tree: First, obtain the statement corresponding to the node, then construct the abstract syntax tree for the corresponding statement. Convert each node in the abstract syntax tree into an index in the dictionary using a predefined dictionary. For each node vi, the node vi is represented as (t1, t2, ..., tj), where j is the number of nodes, and tm, m∈{1, ..., j} is the index of the node in the dictionary. Finally, determine the feature value of the node using a hash function. Step 23) Forming a graph abstract representation of the source code: There are two types of edges between nodes. When analyzing and obtaining the control flow graph of the functions involved, control flow edges between nodes are obtained; when constructing the abstract syntax tree of the statements corresponding to the nodes, abstract syntax edges between nodes are obtained. These two types of edges constitute the edges between nodes in the program graph abstract representation.
3. The graph neural network program defect prediction method based on counterexample training according to claim 1, characterized in that: The graph neural network model mentioned is a gated graph neural network, or GGNN model. Its input is a graph abstract representation of the program source code. The conversion method is to first construct the statement-level control flow graph of the program, where each basic block is a statement, and then represent each statement through an abstract syntax tree.
4. The graph neural network program defect prediction method based on counterexample training according to claim 1, characterized in that: The definition of the counterexample sample is: a new data sample generated by randomly mutating and modifying the statements of samples that the graph neural network model cannot correctly predict in the training dataset.