Source code representation extraction method

By combining BERT, PCA, Multi-way Tree-LSTM and GloVe technologies, combined with attention mechanism BiGRU, the multi-dimensional representation of source code is extracted, which solves the problem of incomplete code characterization information and significantly improves the accuracy and performance of vulnerability detection.

CN118940019APending Publication Date: 2024-11-12SHIHEZI UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411040108.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the source code vulnerability detection, the code characterization information is incomplete and the data embedding effect is not ideal, resulting in a high detection false alarm rate, limiting the effectiveness of vulnerability detection.

Method used

Using BERT model and principal component analysis technology PCA combined with Multi-way Tree-LSTM and GloVe word embedding technology, combined with attention mechanism BiGRU, extract multi-dimensional representation of source code, integrate code lines, code construction, abstract syntax tree subtrees and dependencies. Context features.

Benefits of technology

By more comprehensively reflecting the detailed information of the code and logical structure, the accuracy and performance of vulnerability detection are improved, the false positive rate is reduced, and the effectiveness of vulnerability detection is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940019A_ABST
    Figure CN118940019A_ABST
Patent Text Reader

Abstract

The invention discloses a source code representation extraction method, which comprises the following steps: acquiring a source code data set, and preprocessing the data set; on the basis of a BERT model and a principal component analysis (PCA) technology, feature extraction is carried out on code lines and code structures in the preprocessed data set, and a first basic representation vector and a second basic representation vector are obtained; based on a Multi-way Tree-LSTM (Long Short Term Memory) model, directly extracting subtree features of an abstract syntax tree of a code line to obtain a third basic representation vector; on the basis of a GloVe word embedding technology and a GRU model, dependency context features of the code graph structure are extracted, and a fourth basic representation vector and a fifth basic representation vector are obtained; and combining the obtained basic representation vectors by using an attention mechanism BiGRU to obtain source code representation. Therefore, by adopting the method, multi-aspect information in the source code can be comprehensively reserved, and potential vulnerabilities existing in the code can be captured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of code representation extraction, and in particular to a source code representation extraction method. Background Art

[0002] Source code representation extraction refers to extracting meaningful structures that can represent the intrinsic information and semantics of the code from a piece of source code through static analysis and dynamic analysis. This structure can be processed by machine learning or deep learning models to form a numerical representation that can learn the potential content in the code. This form can be used to perform code-related tasks such as code understanding, code recommendation, and code completion.

[0003] At present, the field of source code vulnerability detection mainly focuses on two categories. One is to use NLP methods to analyze the code based on the text representation of the code, and to extract the code representation through static analysis technology and slicing technology. This method pays more attention to the text information of the code when extracting the code representation, and attempts to analyze the vulnerability representation in the source code from the perspective of text. The other is to use the graph structure of the code intermediate representation to obtain the representation information in the source code. This method mainly considers the logical relationship of the code, and uses static analysis technology and graph embedding technology to obtain the code representation. This method pays more attention to the logical relationship between the vulnerability code when extracting the code representation, and attempts to analyze the vulnerability representation in the source code from the perspective of the graph. However, using the code intermediate representation to represent the source code will still lead to problems such as incomplete information in the representation, unsatisfactory data embedding effect, and high false positive rate of detection, which limits the effectiveness of vulnerability detection.

[0004] Since code representation is the key to vulnerability detection, the quality of code representation determines the effectiveness of vulnerability detection. Therefore, how to obtain an extraction framework that can effectively extract code representation has become the focus of vulnerability detection tasks. Summary of the invention

[0005] The purpose of the present invention is to provide a source code representation extraction method, which combines text representation and an intermediate representation graph structure to represent the source code and can effectively reflect the code content.

[0006] To achieve the above object, the present invention provides a source code representation extraction method, comprising the following steps:

[0007] S1. Obtain source code dataset and preprocess the dataset;

[0008] S2. Based on the BERT model and principal component analysis technology PCA, feature extraction is performed on the code lines and code structures in the preprocessed data set to obtain the first basic representation vector and the second basic representation vector;

[0009] S3, based on the Multi-way Tree-LSTM model, directly extract the abstract syntax tree subtree features of the code line to obtain the third basic representation vector;

[0010] S4. Based on the GloVe word embedding technology and the GRU model, the dependency context features of the code graph structure are extracted to obtain the fourth basic representation vector and the fifth basic representation vector;

[0011] S5. Using the attention mechanism BiGRU, the first basic representation vector, the second basic representation vector, the third basic representation vector, the fourth basic representation vector and the fifth basic representation vector are combined to obtain source code representation.

[0012] Preferably, the preprocessing of the data set is divided into two stages, the first stage includes data cleaning and standardization, and the second stage includes annotation and initial information extraction.

[0013] Preferably, feature extraction is performed on the code structure, including extracting the natural sequence of source code lines based on Tokens, using the BERT architecture combined with a self-attention layer to obtain a representation of the natural sequence of code lines, and using principal component analysis technology PCA for dimensionality reduction.

[0014] Preferably, feature extraction is performed on the code lines, including extracting the abstract syntax tree and control flow graph data information of the source code, extracting code slices of API calls, array usage and pointer usage of the source code based on slicing technology, and obtaining the structural representation of the source code through character matching.

[0015] Preferably, the extraction of dependency context features includes obtaining a code attribute graph of the source code, obtaining a program dependency graph of the code based on the code attribute graph, extracting data dependency context and control dependency context features from the program dependency graph, and vectorizing the extracted features using GloVe word embedding technology.

[0016] Therefore, the present invention adopts the above-mentioned source code representation extraction method, which has the following technical effects:

[0017] (1) Extracting features from source code based on tokens can more comprehensively reflect detailed information about the code, including variable names, keywords, operators, etc., thereby better capturing potential vulnerabilities in the code. In addition, the tokens of code lines can also retain the grammatical structure and contextual information of the code, enabling the model to better understand the meaning and logical structure of the code.

[0018] (2) By extracting the features of the code graph structure, it is possible to mainly extract the dependency, hierarchy, and sequence relationships in the intermediate representation of the source code.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of code line representation extraction in an embodiment of a source code representation extraction method;

[0021] Figure 2 A code line call information extraction process in an embodiment of a source code representation extraction method;

[0022] Figure 3 It is a schematic diagram of dependency context representation extraction in an embodiment of a source code representation extraction method;

[0023] Figure 4 is a schematic diagram of sub-AST representation extraction in an embodiment of a source code representation extraction method;

[0024] Figure 5 is a schematic diagram of BiGRU representation extraction based on attention mechanism in an embodiment of a source code representation extraction method;

[0025] Figure 6 It is a schematic diagram of data set preprocessing in an embodiment of a source code representation extraction method;

[0026] Figure 7 is the performance of different representation extraction methods in a source code representation extraction method embodiment on a FUNDED dataset;

[0027] Figure 8 is a performance of different representation extraction methods in a source code representation extraction method embodiment on a Devign data set;

[0028] Fig. 9 is a performance of different representation extraction methods in an embodiment of a source code representation extraction method on a CVE+Github dataset;

[0029] Fig.10 The present invention is a source code representation extraction method embodiment and the performance of the source code representation extraction method in various graph neural network models. DETAILED DESCRIPTION

[0030] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0031] like Figure 1 As shown, a source code representation extraction method provided by the present invention comprises the following steps:

[0032] (1) Use BERT+PCA to obtain the first basic representation vector F1 and the second basic representation vector F2 of the code line and code structure, as follows:

[0033] The natural sequence of source code lines is extracted based on Tokens, and the representation of the natural sequence of source code lines is obtained by combining the BERT architecture with the self-attention layer.

[0034] Among them, this embodiment uses the pre-trained model of CodeBERT on the CodeSearchNet corpus as the pre-trained model of word embedding; in addition, since BERT outputs 768 dimensions by default, the principal component analysis technology PCA is used to reduce the dimension, so as to obtain more effective data.

[0035] For example, the natural sequence extracted from the code line "if (bind (listenSocket, (struct sockaddr *) & service, sizeof (service)) == SOCKET_ERROR)" is:

[0036] #@"if""bind""var_1""struct""sockaddr""&""var_2""sizeof""var_2""==""SOCKET_ERROR"##

[0037] Each token in the sequence is separated by double quotes, and '#@' is used as the beginning of the code line natural sequence data, and '##' is used as the end of the code line natural sequence data.

[0038] The basic structure of the vulnerability code includes API calls, array usage, and pointer usage in the vulnerable line. In CWE vulnerabilities, incorrect or unsafe API calls can lead to security risks such as data leakage, denial of service, and remote code execution. For example, parameter injection caused by improper validation of input data, resource leakage caused by unhandled exceptions, etc.; incorrect array operations may lead to buffer overflows, pointer out-of-bounds access, memory leaks, etc.; and incorrect pointer operations may lead to null pointer references, wild pointer access, memory leaks, etc. Therefore, the API calls, array usage, and pointer usage in the basic structure of the vulnerability code are also included in the extraction of representations as one of the vulnerability representations, so as to better mine the information in the vulnerability source code.

[0039] Based on SySeVR's syntax slicing technology, the representation of API calls, array usage, and pointer usage is extracted, as follows:

[0040] First, based on the static analysis tool Joern, the source code's abstract syntax tree (AST), control flow graph (CFG) and other data are extracted for program analysis, and the extracted data is stored in the .Joernindex file directory under the Joern directory; then, the Neo4j graph database management system is opened, the data in the .Joernindex file is imported, and the data information in the control flow graph (CFG) is obtained using a script written in Python; then, the slicing technology is used to extract the code slices of API calls, array usage and pointer usage of the entire code; finally, through character matching, the above information is mapped to the corresponding code line to obtain the basic structural representation of the corresponding code line, such as Figure 2 shown.

[0041] For example, the code line "if (bind (var_1, (struct sockaddr *) & var_2, sizeof (var_2)) == SOCKET_ERROR)" extracts the basic structure of the code API call, array use and pointer use as follows:

[0042] #@"bind""(""var_1"",""(""struct""sockaddr*"")""&var_2"",""sizeof""(""var_2"")"")"##

[0043] Among them, "bind", "struct" and "sizeof" in the information are all API calls, while sockaddr* is used as a pointer. Since there is no array, there is no array use. The data is segmented in the same way as natural sequences, and BERT and PCA are used for code representation to obtain a representation of the basic structure of the code line.

[0044] Neo4j used in the above process is an open source NoSQL database management system based on a graph database. It uses a graph structure to store data and provides a data model based on nodes, edges, and attributes. Compared with traditional relational databases, Neo4j can easily handle multiple relationships and connections and is more suitable for handling complex and dynamic data relationships.

[0045] (2) Using GloVe+GRU, obtain the fourth basic representation vector F4 and the fifth basic representation vector F5 based on the dependency context in the code graph structure.

[0046] Among them, the dependencies of the code graph structure include the abstract syntax tree subtree (Sub-AST) and the program dependency graph context (PDG-Context).

[0047] Sub-AST can retain contextual information and structured information related to the code line. Contextual information includes hierarchical information, logical relationships, parallel and selection relationships, etc. It provides the relationship between nodes and the code execution context, which helps to infer the meaning of the code and possible vulnerabilities. Structured information mainly refers to the grammatical structure in the code. Sub-AST represents the grammatical structure of the code in a tree structure, mapping each part of the code to the corresponding node. This structured representation can more clearly understand the organization of the code, making it easier to find potential vulnerability patterns.

[0048] The Program Dependency Graph Context (PDG-Context) mainly extracts its dependencies in the code, including data dependencies and control dependencies. Data dependency means that the value of a variable is used as the input of another variable or expression in the program; if the value of variable A is used to calculate the value of variable B in the program, then we say that variable B has data dependency on variable A, usually represented as a directed edge A->B, where A is called the source node and B is called the target node. It is mainly used to detect abnormal call vulnerabilities in the code, such as user input that has not been fully verified may be directly used for sensitive operations, or improperly handled variable writes may lead to data tampering. Control dependency refers to the dependency of the program execution path. For example, if a program contains an if statement, then the execution path of the program will depend on the value of the conditional expression of the statement. If the condition is true, the execution path of the program will follow the code within the if block, otherwise the program will skip the if block and execute the else block or the following code. The if block has a control dependency on its conditional expression. The control dependency is usually represented as a directed edge C->D, where C is called the source node and D is called the target node. The control dependency mainly detects logical errors in the code, such as incorrect loop termination conditions in conditional statements.

[0049] Therefore, the basic representation of dependency context includes data dependency context (DDC) and control dependency context (CDC), as follows:

[0050] First, the code property graph (CPG) of the source code is obtained through the static analysis tool Joern; then, the program dependency graph (PDG) of the code is obtained based on the code property graph (CPG); then, based on the API interface of Joern and the data flow edges and control flow edges in PDG, the data dependency context DDC and control dependency context CDC of the code line are extracted from the program dependency graph (PDG); finally, the word embedding technology GloVe is used in combination with the pre-trained model for vectorization to obtain the dependency context representation of the code, such as Figure 3shown.

[0051] In the process of GloVe vectorization, since the size of each line of program-dependent information code block is different, the length of the corresponding representation will be different. Therefore, zero padding technology is used, that is, adding a 0-valued element at the end of the input sequence to obtain a fixed-length input, and a masking layer is added to dynamically instruct the model to ignore the information of the padding part, so that it can focus on the calculation of the effective part, thereby improving the efficiency and accuracy of the model.

[0052] (3) Extract the third basic representation vector F3 of the abstract syntax tree subtree (Sub-AST) of the code line, as follows:

[0053] First, we use Joern to get the global subgraph of the code, and then get the abstract syntax tree subtree (Sub-AST) of the code line. Then, according to the word element of the code line, we add the information of the last node in the previous subtree to the abstract syntax tree of the current line through character matching to get the final abstract syntax tree subtree (Sub-AST). Finally, we use Multi-way Tree-LSTM to convert Sub-AST into a representation vector, as shown in the following example: Figure 4 shown.

[0054] (4) Using the attention mechanism BiGRU, the obtained basic representations F1, F2, F3, F4 and F5 are merged into the source code representation.

[0055] Among them, BiGRU based on the attention mechanism can dynamically assign attention weights according to the importance of the input, and by calculating the attention score of each time step, the weighted sum of the forward and backward hidden states is added to generate the final representation vector. This method can give more weight to important representations, thereby improving the expressiveness and performance of the vulnerability detection model. Figure 5 shown.

[0056] This embodiment uses existing public datasets (Fan, Reveal, Devign, and FUNDED) and self-collected datasets (CVE+Github), as shown in Table 1.

[0057] Table 1 Dataset description

[0058] Dataset Fan Reveal Devign FUNDED CVE+Github Vulnerability Dataset 11,814 1,660 10,067 49,600 12,165 Vulnerability-free dataset 148,746 16,500 12,294 49,600 12,165 Proportion (Vulnerability: No Vulnerability) 1:15 1:10 1:1.2 1:1 1:1

[0059] The preprocessing of the dataset is divided into two stages: the first stage is data cleaning and standardization operations. In this stage, it is necessary to remove the content with unclear representation and retain the valid data for subsequent research. At the same time, standardization operations are performed to make the data representation more obvious; the second stage is annotation and initial information extraction. The annotation operation is mainly aimed at the datasets Fan and CVE+Github with vulnerable lines, annotating the vulnerable lines for subsequent vulnerability location research, and initial information extraction mainly processes the code through static analysis tools to extract information such as the intermediate representation of the code to form a data set that can be input into the representation extraction framework, such as Figure 6 shown.

[0060] Among them, the operations in the first phase are mainly aimed at source code files. Data cleaning mainly solves problems such as data duplication, chaotic logical hierarchy and interference from irrelevant information in the data set. Operations include format normalization, comment and blank line processing, code snippet extraction and coding error correction, so as to obtain clean, parsable and consistent code data for unified input and convenient representation extraction; standardization is to map the custom variable names and function names of the code to unified variable names (var_xx) and function names (fun_xx), etc. xx represents the sequential number. This step is completed through Python scripts, and standardized codes are used to distinguish code lines.

[0061] In the second stage, the annotation and initial information extraction operations are mainly used to generate CSV files. Annotation refers to recording the line number and label of the vulnerable line in the source code file. This information is used for subsequent code detection research; initial information extraction includes extracting the natural sequence of the code, the basic representation (call) slice of the code, the intermediate representation of the code (AST, PDG, CFG, etc.) and the basic information of the vulnerability data, and the obtained information is stored in CSV. Among them, the intermediate representation is used to obtain the implicit dependencies in the code.

[0062] The preprocessed data sets are divided into training, validation and test sets in a ratio of 6:1:3, and the model training iteration number epoch is set to 800, the learning rate lr is set to 0.0001, the random seed is set to 1234, the dropout is set to 0.3, the representation input dimension is set to 128, and the dimension of the hidden state is set to 256. In the gated graph recurrent layer of GGNN, the number of repeated steps is 8, and the ReLU activation function is used to process the results of each layer. At the same time, the maximum pooling layer is used in the third convolutional layer to complete the representation extraction, and the Adam optimizer is used to avoid overfitting, so as to verify the effectiveness of the code representation extraction method in this embodiment.

[0063] Verification Example 1

[0064] This example compares the existing representation construction methods used by Devign and Reveal to verify the performance of the source code representation extraction method provided in this example in vulnerability detection on different data sets, and evaluates it through evaluation indicators such as accuracy (ACC), F1-score parameter (F1), precision (P), recall (R), and AUC (Area Under the Curve). Figure 7-9 shown.

[0065] Figure 7-9 The results show that the source code representation extraction method provided in this embodiment shows high accuracy and F1-score parameters in the FUNDED dataset, the Devign dataset, and the CVE+Github dataset. Therefore, the method of the embodiment has a better performance than the existing Reveal and Devign representation extraction frameworks.

[0066] Verification Example 2

[0067] On the CVE+Github dataset, five graph neural network models, GCN, GraphSAGE, GGNN, GAT, and SGAT, are designed for vulnerability detection to verify the performance of the method of the embodiment under each graph neural network model. Fig.10 shown.

[0068] from Fig.10 It can be seen that the source code representation extraction method adopted in this embodiment performs best in SGAT, with the highest ACC reaching 0.73 and the F1 value reaching 0.72, while GCN performs the worst, with the ACC value of 0.66 and the F1 value of 0.67 under the GCN model.

[0069] Verification Example 3

[0070] The SGAT model is used as the network framework, and the parameters Epoch is set to 600, the input dimension is 128, the hidden layer is 256, the output dimension is 2, the learning rate lr is 0.001, and the dropout_rate is 0.3. After multiple experiments, the number of convolutional layers is set to 3, and the number of multi-head attention mechanism headers is set to 6. The Relu function is used for the results of each layer to prevent overfitting, and Adam is used as the experimental optimizer to verify the influence of each representation on the results, as shown in Table 2.

[0071] Table 2 The influence of each characterization on the results

[0072]

[0073]

[0074] As can be seen from Table 2, the impact of the text representation structures CNS and CBC on ACC is between 2.6% and 4%. On the dataset Fan, the impact of the CNS and CBC structures is the smallest, which is 2.6%, and on the dataset Devign, the impact is the largest, which is 4%. The impact of the text representation structures CNS and CBC structures on the F1 evaluation index is between 2.8% and 7.9%, with the smallest impact on the Fan dataset, which is 2.8%, and the largest impact on the CVE+Github dataset, which is 7.9%.

[0075] The impact of Sub-AST basic representation on ACC is between 7.8% and 9.2%, with the smallest impact on the FUNDED dataset being 7.8% and the largest impact on Devign being 9.2%. The impact of Sub-AST basic representation on F1 parameters is between 4.7% and 8.9%, with the largest impact on the Fan dataset being 8.9% and the smallest impact on the FUNDED dataset being 4.7%.

[0076] In addition, using different models to extract Sub-AST representations will have different effects on the results. Among them, using Multi-way Tree-LSTM to process Sub-AST can improve ACC by about 1.0%-1.6% and F1 parameters by about 0.9%-2.5%. Therefore, using Multi-way Tree-LSTM to process AST is more effective than Tree-LSTM.

[0077] Therefore, the present invention adopts the above-mentioned source code representation extraction method, which can represent the text features of the code and the graph structure formed by the intermediate representation, and effectively reflect the text feature content of the code.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A source code representation extraction method, characterized in that it includes the following steps: S1. Obtain source code dataset and preprocess the dataset; S2. Based on the BERT model and principal component analysis technology PCA, feature extraction is performed on the code lines and code structures in the preprocessed data set to obtain the first basic representation vector and the second basic representation vector; S3, based on the Multi-way Tree-LSTM model, directly extract the abstract syntax tree subtree features of the code line to obtain the third basic representation vector; S4. Based on the GloVe word embedding technology and the GRU model, the dependency context features of the code graph structure are extracted to obtain the fourth basic representation vector and the fifth basic representation vector; S5. Using the attention mechanism BiGRU, the first basic representation vector, the second basic representation vector, the third basic representation vector, the fourth basic representation vector and the fifth basic representation vector are combined to obtain source code representation.

2. A source code representation extraction method according to claim 1, characterized in that the preprocessing of the data set is divided into two stages, the first stage includes data cleaning and standardization, and the second stage includes annotation and initial information extraction.

3. A source code representation extraction method according to claim 1, wherein the representation is to extract features from code structures, including extracting natural sequences of source code lines based on Tokens, obtaining representations of natural sequences of code lines using a BERT architecture combined with a self-attention layer, and performing dimensionality reduction using principal component analysis (PCA).

4. A source code representation extraction method according to claim 1, wherein the representation is to perform feature extraction on code lines, including extracting the abstract syntax tree and control flow graph data information of the source code, extracting code slices of API calls, array usage and pointer usage of the source code based on slicing technology, and obtaining the structural representation of the source code through character matching.

5. According to the source code representation extraction method of claim 1, the extraction of dependency context features includes obtaining a code attribute graph of the source code, obtaining a program dependency graph of the code based on the code attribute graph, extracting data dependency context and control dependency context features from the program dependency graph, and vectorizing the extracted features using GloVe word embedding technology.