Source code tracing method, system and readable storage medium based on graph neural network

Abstract

This invention discloses a source code traceability method, system, and readable storage medium based on a graph neural network. The method includes the following steps: extracting a source code comparison database; performing vectorization preprocessing on the source code; constructing a source code feature representation model based on a graph convolutional neural network; calculating source code similarity based on the graph edit distance of the source code feature representation model; and tracing the source code based on the source code similarity. This invention addresses the problems in related technologies that lack evaluation of source code content information and logical relationships, and the inability to accurately trace the source code.

Description

Technical Field

[0001] The present invention belongs to the field of software security technology, and in particular relates to a source code tracing method, system and readable storage medium based on graph neural network. Background Art

[0002] With the continuous advancement of science and technology and the growing popularity of open source on the internet, developers can quickly access the code snippets they need through open source code repositories such as GitHub, Code Cloud, and Stack Overflow. By copying, modifying, and referencing existing open source code within their own projects, they can significantly improve project development efficiency. Open source software has brought significant convenience to the information technology industry, improving development efficiency and reducing costs. However, this has also led to problems with code reuse and plagiarism. First, code plagiarism poses a significant threat to software intellectual property protection, negatively impacting individuals and companies, and even constituting a criminal offense. Second, copying and pasting code without considering the data and control flow structure of a software program can reduce program stability, potentially leading to the spread of code defects and code redundancy. Third, given the urgent need for domestic software development, open source code reuse and plagiarism can present certain obstacles, increasing the difficulty and uncontrollability of code maintenance. Therefore, software code traceability has become a key research direction in software supply chain security.

[0003] Existing software code traceability methods mainly use the longest common subsequence algorithm or cosine similarity algorithm to perform similarity analysis on the features of a single bytecode aggregate line. However, they lack the evaluation of the content information and logical relationships of the source code, resulting in inaccurate similarity evaluation results and the inability to achieve accurate traceability.

[0004] In order to finely characterize the source code and accurately evaluate the source code similarity and thus accurately trace the source code, a source code tracing method, system and readable storage medium based on graph neural network are proposed. Summary of the Invention

[0005] The embodiments of the present invention propose a source code tracing method, system and readable storage medium based on a graph neural network, so as to at least solve the problems in related technologies of lack of evaluation of source code content information and logical relationships and inability to accurately trace the source.

[0006] According to one embodiment of the present invention, a source code tracing method based on a graph neural network is proposed, comprising:

[0007] Extract source code and compare database;

[0008] Perform vectorization preprocessing on the source code;

[0009] Construct a source code feature representation model based on graph convolutional neural network;

[0010] Calculate source code similarity based on the graph edit distance of the source code feature representation model;

[0011] Trace the source code based on the source code similarity.

[0012] In an exemplary embodiment, extracting a source code comparison database includes the steps of:

[0013] Projects that obtain source code from open source websites; each project contains multiple code files, and each source code file contains one or more functions;

[0014] Build a source code comparison database, including any one or a combination of web sites, crawlers, file processing and data logging.

[0015] In an exemplary embodiment, the vectorization preprocessing of the source code includes the steps of:

[0016] Generate an abstract syntax tree for each function in each code file;

[0017] Classify node types according to node classes and / or methods and / or statements;

[0018] Vectorize and encode the node content of the abstract syntax tree according to the node type to form a node vector;

[0019] Use the depth-first search algorithm to describe the edge structure relationship of the abstract syntax tree to form an edge structure vector;

[0020] The vectorized representation of the source code is constructed based on the node vector and the edge structure vector.

[0021] In an exemplary embodiment, dividing the node types according to the node classes and / or methods and / or statements comprises the steps of:

[0022] Calculate class similarity based on homology and / or functional relevance of node class structures;

[0023] Method similarity is calculated based on the path similarity and / or result consistency of the node method structure;

[0024] Calculate sentence similarity based on the consistency of node sentence structure;

[0025] Calculating node type relevance based on class similarity and / or method similarity and / or statement similarity;

[0026] Nodes with a node type correlation greater than a preset threshold are classified as the same node type.

[0027] In an exemplary embodiment, the construction of a source code feature representation model based on a graph convolutional neural network includes the following steps:

[0028] Construct a one-dimensional graph neural network model with node features based on a one-dimensional convolutional neural network;

[0029] Construct an edge feature 2D graph neural network model based on a 2D convolutional neural network;

[0030] The one-dimensional graph neural network model of node features and the two-dimensional graph neural network model of edge features are aggregated to generate a source code feature representation model based on graph convolutional neural network.

[0031] In an exemplary embodiment, the one-dimensional graph neural network model for node features constructed based on the one-dimensional convolutional neural network is a node feature extraction layer constructed based on the one-dimensional convolutional neural network, which characterizes the features of the one-dimensional node vector; the two-dimensional graph neural network model for edge features constructed based on the two-dimensional convolutional neural network is an edge feature extraction layer constructed based on the two-dimensional convolutional neural network, which characterizes the features of the two-dimensional edge structure vector; the one-dimensional graph neural network model for node features and the two-dimensional graph neural network model for edge features are aggregated to generate a source code feature representation model based on the graph convolutional neural network, which constructs an aggregation layer based on the fully connected layer, and aggregates the node features and edge features to obtain the AST tree graph feature representation of the source code.

[0032] In an exemplary embodiment, the method of calculating source code similarity based on the graph edit distance of the source code feature representation model includes the steps of:

[0033] Calculate the graph edit distance between different functions based on the source code feature representation model based on graph convolutional neural network;

[0034] Calculate the graph edit distance matrix between source codes based on the function structure of the source code;

[0035] The source code similarity is calculated based on the graph edit distance matrix between source codes.

[0036] In an exemplary embodiment, tracing the source code based on the source code similarity includes the steps of:

[0037] Calculate the similarity between the source code and the source code of each function in the open source library;

[0038] Calculate the maximum value of source code similarity and take the function corresponding to the maximum value as the most similar function;

[0039] Identify the most similar engineering projects based on the most similar functions, thus achieving source code traceability.

[0040] A computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the above method.

[0041] According to another embodiment of the present invention, a source code traceability system based on a graph neural network is provided, comprising:

[0042] processor;

[0043] Memory;

[0044] as well as

[0045] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs causing the computer to perform the above method.

[0046] The graph neural network-based source code tracing method, system, and readable storage medium of the present invention have the following advantages:

[0047] (1) Converting the source code into an abstract syntax tree and dividing the node types according to the node class and / or method and / or statement can effectively improve the representation accuracy of the source code compared to the traditional technical solution of only analyzing the bytecode line.

[0048] (2) The node contents of the abstract syntax tree are vectorized and encoded according to the node type to form a node vector. The edge structure relationship of the abstract syntax tree is described using a depth-first search algorithm to form an edge structure vector, which is then used to form a vectorized representation of the source code. Compared with the traditional technical solution that only analyzes bytecode lines, this can fully represent the content information and logical relationships of the source code.

[0049] (3) For node features, a one-dimensional graph neural network is used to extract node feature information; for the adjacency matrix between nodes, a two-dimensional graph neural network is used to extract edge features between nodes. Then, through the aggregation network, the graph feature representation of the AST tree of the source code is obtained and the source code similarity is calculated based on the graph edit distance. Compared with the traditional technical solution of calculating similarity based on the longest common subsequence algorithm or the cosine similarity algorithm, this can comprehensively and effectively evaluate the similarity between the fine features of the source code and improve the accuracy of source code traceability. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of a source code tracing method based on a graph neural network according to an embodiment of the present invention;

[0051] Figure 2 is a flowchart of sub-step S01 of an embodiment of the present invention;

[0052] Figure 3 is a flowchart of sub-step S02 of an embodiment of the present invention;

[0053] Figure 4 is a flowchart of sub-step S022 of an embodiment of the present invention;

[0054] Figure 5is a flowchart of sub-step S03 of an embodiment of the present invention;

[0055] Figure 6 is a flowchart of sub-step S04 of an embodiment of the present invention;

[0056] Figure 7 is a flowchart of sub-step S05 of an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of the structure of a source code tracing system based on a graph neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the invention, but are not intended to limit the present invention in any form. It should be noted that those skilled in the art may make several changes and modifications without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0059] A source code tracing method based on graph neural network according to an embodiment of the present invention is shown in the flowchart Figure 1 As shown, the steps include:

[0060] Step S01, extracting the source code comparison database;

[0061] Step S02: performing vectorization preprocessing on the source code;

[0062] Step S03: constructing a source code feature representation model based on graph convolutional neural network;

[0063] Step S04: Calculate source code similarity based on the graph edit distance of the source code feature representation model;

[0064] Step S05: Tracing the source code based on the source code similarity.

[0065] In an exemplary embodiment, the step S01 is as shown in the flow chart. Figure 2 Shown, including:

[0066] Step S011: Obtain source code engineering projects from open source websites; each engineering project includes multiple code files, and each source code file includes one or more functions;

[0067] Step S012: constructing a source code comparison database, including any one or a combination of Web sites, crawlers, file processing, and data records.

[0068] In this embodiment, a Java source code project is obtained from an open source website through a crawler to build a Java source code library. Each project contains multiple Java code files, and each source code file contains one or more functions. A source code comparison database is built, which includes any one or more combinations of websites, crawlers, file processing and data recording.

[0069] In an exemplary embodiment, the step S02 is as shown in the flowchart. Figure 3 Shown, including:

[0070] Step S021: Generate an abstract syntax tree for each function in each code file;

[0071] Step S022: classify the node types according to the node classes and / or methods and / or statements;

[0072] Step S023: vectorize and encode the node content of the abstract syntax tree according to the node type to form a node vector;

[0073] Step S024: Use a depth-first search algorithm to describe the edge structure relationship of the abstract syntax tree to form an edge structure vector;

[0074] Step S025: construct a vectorized representation of the source code based on the node vectors and the edge structure vectors.

[0075] In this example, Java Lang is used to perform a deep analysis of the Java source code. For each function in each code file, an abstract syntax tree (AST) is generated. This tree structure is constructed based on the code logic, representing the syntax and structure of the code. Each node in the tree represents a different grammatical structure, such as a class, method, or statement, and the edges in the tree represent the execution order between classes, methods, or statements.

[0076] In an exemplary embodiment, the sub-step S022 is to classify the node types according to the node classes and / or methods and / or statements, as shown in the flowchart. Figure 4 Shown, including:

[0077] Step S0221: Calculate class similarity based on the homology and / or functional relevance of the node class structure;

[0078] Step S0222: Calculate method similarity based on the path similarity and / or result consistency of the node method structure;

[0079] Step S0223, calculating sentence similarity based on the consistency of the node sentence structure;

[0080] Step S0224: Calculate node type similarity based on class similarity and / or method similarity and / or statement similarity;

[0081] Step S0225: Classify nodes whose node type similarity is greater than a preset threshold into the same node type.

[0082] In this embodiment, the class similarity is calculated based on the homology and / or functional relevance of the node class structure, which is: calculating the class similarity based on the positive correlation between the homology of the node class structure (calculated based on the overlap rate of the library to which the class belongs) and the class similarity, calculating the class similarity based on the positive correlation between the functional relevance of the node class structure (calculated based on the ratio of the number of times two class structures are used simultaneously to the total number of times the class structure is used) and the class similarity, or calculating the class similarity based on the positive correlation between the homology and functional relevance of the node class structure and the class similarity, and the class similarity is represented by the variable e.

[0083] The method similarity is calculated based on the path similarity and / or result consistency of the node method structure, which is: calculating the method similarity based on the positive correlation between the similarity of the function implementation path of the node method structure (calculated based on the overlapping rate of the algorithm steps) and the method similarity, calculating the method similarity based on the positive correlation between the consistency of the function implementation results of the node method structure (evaluated based on whether the algorithm results are the same and the length of the algorithm for the implementation results) and the method similarity, or calculating the method similarity based on the positive correlation between the similarity of the function implementation path of the node method structure and the consistency of the function implementation results and the method similarity, and the method similarity is represented by the variable w.

[0084] The calculation of sentence similarity based on the consistency of the sentence structure of the nodes is to evaluate the consistency of the sentence structure based on the proportion of the same sentence structure between the nodes and calculate the sentence similarity based on the positive correlation between the consistency of the sentence structure and the sentence similarity. The sentence similarity is represented by the variable y.

[0085] The node type similarity is calculated based on the class similarity and / or method similarity and / or statement similarity according to the positive correlation between the class similarity and / or method similarity and / or statement similarity and the node type similarity between different nodes. The node type similarity is represented by the variable x.

[0086] A1 to A7 in Table A represent different implementation methods for calculating node type similarity. For ease of expression, the node type similarity x in Table A represents the node type similarity between two nodes, and the class similarity e, method similarity w, and statement similarity y are calculated using the method described in any of the above implementation methods.

[0087] Table A Different implementation methods for calculating node type similarity

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] In step S0225, nodes whose node type similarity is greater than a preset threshold are classified as the same node type. The preset threshold X is obtained by training based on the source code type, library type, or node type. In this embodiment, the preset threshold X=0.7, and nodes whose node type similarity x>0.7 are classified as the same node type. According to the node features in one embodiment, the nodes are divided into 25 types, including {"FunctionDef", "AsyncFunctionDef", "ClassDef", "Return", "Delete", "Assign", "AugAssign", "AnnAssign", "For", "AsyncFor", "While", "If", "with", "AsyncWith", "Raise", "Try", "Assert", "Import", "ImportFrom", "Global", "Nonlocal", "Expr", "pass", "Break", "Continue", "OtherElse"}.

[0096] In this embodiment, according to the 25 node types, the node content of the AST tree is first one-hot vectorized and encoded, and the node is converted into a vector V, which is the node vector V of the function; then, a depth-first search algorithm is used to describe the edge structure relationship of the function tree and generate a variable structure vector E; the vectorized representation of the source code is constructed based on the node vector V and the edge structure vector E, which is denoted as G = (V, E).

[0097] In an exemplary embodiment, the step S03 is as shown in the flowchart. Figure 5 Shown, including:

[0098] Step S031: construct a node feature one-dimensional graph neural network model based on a one-dimensional convolutional neural network;

[0099] Step S032: construct an edge feature two-dimensional graph neural network model based on the two-dimensional convolutional neural network;

[0100] Step S033: The node feature one-dimensional graph neural network model and the edge feature two-dimensional graph neural network model are aggregated to generate a source code feature representation model based on a graph convolutional neural network.

[0101] The one-dimensional graph neural network model for node features constructed according to the one-dimensional convolutional neural network is a node feature extraction layer constructed based on the one-dimensional convolutional neural network, which characterizes the features of the one-dimensional node vector; the two-dimensional graph neural network model for edge features constructed according to the two-dimensional convolutional neural network is an edge feature extraction layer constructed based on the two-dimensional convolutional neural network, which characterizes the features of the two-dimensional edge structure vector; the one-dimensional graph neural network model for node features and the two-dimensional graph neural network model for edge features are aggregated to generate a source code feature representation model based on the graph convolutional neural network, which constructs an aggregation layer based on the fully connected layer, and aggregates the node features and edge features to obtain the AST tree graph feature representation of the source code.

[0102] In this embodiment, a node feature extraction layer is first constructed based on a one-dimensional convolutional neural network to characterize the features of the one-dimensional node vector V. The one-dimensional convolutional neural network includes an input layer, L convolutional layers, and L pooling layers, where L is a preset convolutional neural network parameter. The convolution layer learns the local structural features of the node by inner product with the convolution kernel; the pooling uses the maximum pooling function to learn the global feature representation of the node. The node vector V is input to the input layer, and after calculation by L convolutional layers and L pooling layers, the node features are extracted and expressed as f v =ReLU(Convolution1D(V l )), where ReLU(·) represents the nonlinear activation function, K l represents the convolution kernel of layer l, b l Indicates the offset.

[0103] Next, an edge feature extraction layer is constructed based on a two-dimensional convolutional neural network to characterize the features of the two-dimensional edge structure vector E. The two-dimensional convolutional neural network consists of an input layer, four convolutional layers, and four pooling layers. The convolutional layer learns the local structural features of the edge by performing inner product with the convolution kernel; the pooling layer uses the maximum pooling function to learn the global feature representation of the edge. The node vector E is input to the input layer. After calculations through the four convolutional layers and four pooling layers, the edge features are extracted and represented as f E =ReLU(Convolution2D(E l )).

[0104] Finally, an aggregation layer is built based on the fully connected layer to aggregate the node features and edge features output from the first two parts to obtain the graph feature representation of the AST tree of the Java source code, which is represented as f ast =concatenate(f V ,f E), where concatenate(·) represents the aggregation function of node features and graph features.

[0105] In an exemplary embodiment, the step S04 is as shown in the flowchart. Figure 6 Shown, including:

[0106] Step S041: Calculate the graph edit distance between different functions according to the source code feature representation model based on the graph convolutional neural network;

[0107] Step S042: Calculate the graph edit distance matrix between source codes according to the function structure of the source codes;

[0108] Step S043: Calculate the source code similarity based on the graph edit distance matrix between the source codes.

[0109] In this embodiment, it is assumed that the feature matrix of the mth function in the Java source code to be compared is extracted by the convolutional neural network as f m,w , the feature matrix of the i-th function code of the n-th project in the Java source code library extracted by the convolutional neural network is Computing graph edit distance in, Represents all features f from the graph m,w Convert to The complete editing path of c(e j ) represents the editing operation e j The cost, λ min represents the path with the smallest sum of costs among all fully edited paths.

[0110] The method of calculating the graph edit distance matrix between source codes based on the function structure of the source code includes the following steps: calculating the graph edit distance between each function in the Java source code to be compared and each function in the engineering code in the source code library; storing the graph edit distance in the form of a two-dimensional matrix based on the function structure of the source code, that is, the graph edit distance matrix between source codes, wherein the mth row represents the graph edit distance between the mth function in the Java source code to be compared and each function in each source code library.

[0111] The source code similarity is calculated based on the negative correlation between the graph edit distance matrix between the source code in the project file to be compared and the graph edit distance of the mth function in the nth project in the source code library and the source code similarity:

[0112] In an exemplary embodiment, the step S05 is as shown in the flowchart. Figure 7 Shown, including:

[0113] Step S051: Calculate the similarity between the source code and the source code of each function in the open source library;

[0114] Step S052: Calculate the maximum value of the source code similarity and use the function corresponding to the maximum value as the most similar function;

[0115] Step S053: Identify the most similar engineering project based on the most similar function to achieve traceability of the function and project.

[0116] In this embodiment, the source code similarity between the source code and each function in the open source library is calculated according to the calculation method of the source code similarity in the above embodiment, and a similarity matrix is ​​formed. The maximum value of a row (for example, the mth row) in the similarity matrix is ​​calculated, and the function corresponding to the maximum value (the mth function) is the most similar function. The most similar engineering projects are identified based on the most similar functions, and the most similar functions and the most similar engineering projects corresponding to the source code are identified to achieve source code traceability.

[0117] In another preferred embodiment, the most similar function for each function in each project, along with the project name and download URL of the project to which the most similar function belongs, is integrated into a report and provided to the user. The Java Source Code Similarity and Tracing Report lists the most similar functions between the functions in the project under test and the open source project, along with the similarity values ​​and download sources of the similar functions.

[0118] A computer-readable storage medium according to an embodiment of the present invention stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the above method.

[0119] A source code traceability system based on graph neural network according to an embodiment of the present invention is shown in the following diagram: Figure 8 Shown, including:

[0120] processor;

[0121] Memory;

[0122] as well as

[0123] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs causing the computer to perform the above method.

[0124] Of course, those skilled in the art should realize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. As long as they are within the scope of the present invention, any changes or modifications to the above embodiments will fall within the scope of protection of the present invention.

Claims

1. A source code tracing method based on graph neural network, characterized in that: include: Extract source code and compare database; Perform vectorization preprocessing on the source code; The vectorization preprocessing of the source code includes the steps of: generating an abstract syntax tree for each function in each code file; Node types are classified according to the node class and / or method and / or statement; the node content of the abstract syntax tree is vectorized and encoded according to the node type to form a node vector; the edge structure relationship of the abstract syntax tree is described using a depth-first search algorithm to form an edge structure vector; the node vector and the edge structure vector constitute a vectorized representation of the source code; Constructing a source code feature representation model based on a graph convolutional neural network; said constructing a source code feature representation model based on a graph convolutional neural network comprises the steps of: constructing a node feature one-dimensional graph neural network model based on a one-dimensional convolutional neural network; constructing an edge feature two-dimensional graph neural network model based on a two-dimensional convolutional neural network; The one-dimensional graph neural network model of node features and the two-dimensional graph neural network model of edge features are aggregated to generate a source code feature representation model based on graph convolutional neural network; Calculating source code similarity based on the graph edit distance of the source code feature representation model; calculating source code similarity based on the graph edit distance of the source code feature representation model, comprising the steps of: calculating the graph edit distance between different functions based on the source code feature representation model based on a graph convolutional neural network; calculating the graph edit distance matrix between source codes based on the function structure of the source codes; and calculating source code similarity based on the graph edit distance matrix between source codes; Trace the source code based on the source code similarity.

2. The source code tracing method based on graph neural network according to claim 1 is characterized in that: The method of extracting the source code comparison database includes the following steps: Projects that obtain source code from open source websites; each project contains multiple code files, and each code file contains one or more functions; Build a source code comparison database, including any one or a combination of web sites, crawlers, file processing and data logging.

3. The source code tracing method based on graph neural network according to claim 1 is characterized in that: The method of dividing the node types according to the node classes and / or methods and / or statements comprises the steps of: Calculate class similarity based on homology and / or functional relevance of node class structures; Method similarity is calculated based on the path similarity and / or result consistency of the node method structure; Calculate sentence similarity based on the consistency of node sentence structure; Calculating node type relevance based on class similarity and / or method similarity and / or statement similarity; Nodes with a node type correlation greater than a preset threshold are classified as the same node type.

4. The source code tracing method based on graph neural network according to claim 1 is characterized in that: The one-dimensional graph neural network model for node features constructed according to the one-dimensional convolutional neural network is a node feature extraction layer constructed based on the one-dimensional convolutional neural network, which characterizes the features of the one-dimensional node vector; the two-dimensional graph neural network model for edge features constructed according to the two-dimensional convolutional neural network is an edge feature extraction layer constructed based on the two-dimensional convolutional neural network, which characterizes the features of the two-dimensional edge structure vector; the one-dimensional graph neural network model for node features and the two-dimensional graph neural network model for edge features are aggregated to generate a source code feature representation model based on the graph convolutional neural network, which constructs an aggregation layer based on the fully connected layer, and aggregates the node features and edge features to obtain the AST tree graph feature representation of the source code.

5. The source code tracing method based on graph neural network according to claim 1 is characterized in that: The source code tracing according to the source code similarity includes the following steps: Calculate the similarity between the source code and the source code of each function in the open source library; Calculate the maximum value of source code similarity and take the function corresponding to the maximum value as the most similar function; Identify the most similar engineering projects based on the most similar functions, thus achieving source code traceability.

6. A computer-readable storage medium storing a computer program for electronic data exchange, wherein: The computer program enables a computer to execute the method according to any one of claims 1 to 5.

7. A source code traceability system based on graph neural network, characterized by include: processor Memory; as well as One or more programs, wherein the one or more programs are stored in a memory and configured to be executed by the processor, the programs causing the computer to perform the method according to any one of claims 1 to 5.

Images