A code summary generation method and device based on structural attention

Abstract

The present invention discloses a code summary generation method and device based on structural attention, which are specifically as follows: S1, splitting the nodes of an original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree; S2, based on the split abstract syntax tree, obtaining original edges, constructing control flow edges and data flow edges, and designing an adjacency matrix of the original edges; S3, performing joint operations on the adjacency matrix of the original edges and the control flow, and on the adjacency matrix of the original edges and the data flow, respectively, to obtain a structure-based attention score matrix; S4, constructing a code summary model based on structural attention using the structure-based attention score matrix, a sequence of pre-order traversal, an edge relationship matrix and a hierarchy matrix; S5, generating a code summary using the constructed code summary model based on structural attention. The present invention can learn the features of the source code more comprehensively, thereby better generating a code summary.

Description

Technical Field

[0001] The present invention relates to the technical field of code summarization, and more particularly to a method and device for generating code summarization based on structural attention. Background Art

[0002] Since the format of source code and its summary is a pair of sequences, this task was initially regarded as a sequence-to-sequence (Seq2Seq) task and equipped with MNT (machine translation) technology. In the early days, researchers regarded source code as a sequence of tokens and introduced related methods on this basis. Because it is believed that program sequences may contain some patterns that can be learned like natural languages. So Iyer et al. proposed a method called CODE-NN, which uses attention mechanism and long short-term memory (LSTM) to summarize C# source code. Allamanis et al. tried to combine convolutional neural network (CNN) with attention mechanism. Although this method has opened up the application of deep learning technology in the field of code summarization, there is still more information to be discovered in source code sequences. Another research team, Hu et al., noticed that the API in the source code sequence is meaningful to developers' understanding. Therefore, they designed a TL-CoSum framework to transfer the learned API knowledge to the code summary task. Zheng et al. proposed a new attention mechanism that utilizes key sentence information. All of the above methods use models based on recurrent neural networks, which makes it difficult to capture long-distance dependencies. In order to solve this problem, Ahmad et al. used the Transformer model to achieve this task. Since all the above studies lack modeling of non-sequential information, relative position encoding is added to the model.

[0003] Many researchers have begun to try to learn information from the Abstract Syntax Tree (AST), a tree-like data structure used to represent the grammatical structure of code in a programming language, based on the scarcity of structural information noticed by Ahmad et al. Hu et al. explored ways to capture grammatical information from AST and designed a structure-based traversal method (SBT) to protect the structure from being destroyed by adding brackets during serialization. Alon et al. represented AST as a set of paths from a terminal node to another terminal node or from a root node to a terminal node, and its performance was better than the research results of Iyer et al. and Allamanis et al. This also illustrates the importance of grammatical information when the model interprets the program. However, Hu et al.'s method of linearizing AST by adding special characters to maintain the structure will definitely extend the input length, and there is no other way for the model to learn these inputs better. Wu et al. pointed out that due to the above reasons, SBT may not improve performance in some cases. Therefore, Wu et al. designed a structure-induced Transformer (SiT) to supplement more about the relationship between data flow and control flow and convert it into a mask matrix. The approach is to take the AST pre-sorted traversal sequence as input and use the mask to induce the Transformer to focus on tokens related to the model data flow, control flow, and syntax. Tang et al. and Gong et al. implemented this idea in a similar way. The former focused on ancestor-descendant and sibling relationships, while the latter introduced a matrix to record the relative distances of AST node pairs. Choi et al. considered other AST pre-sorting of similar code snippet sequences and corresponding summaries and fused them in the decoding stage.

[0004] Therefore, other researchers have explored ways to fuse these two representations or other more different granularity representations. Wan et al. combined the two representations by concatenating the attention scores of LSTM into a reinforcement learning framework early on. But there are two points that need to be improved: one is the method of learning AST, which has always been a major challenge for this task; the other is the method of fusing the two representations together. To overcome these challenges, Shi et al. divided the AST into many subtrees according to the level and encoded each node of the subtree through RNN. The encoded AST features and source code tokens are learned one by one by the cross-attention module in the decoder. Shi et al.'s method seems to be able to maintain the characteristics of the layer and connect the two different representations during decoding. Gao et al. created a matrix of token (sub-token relationship), statement, and data flow hierarchical relationships and applied it to different attention heads when calculating in Transformer. Son et al. believed that it was challenging to extract structural features from tokens and proposed to learn structural features from the program's statement-level representation, the program dependency graph (PDG). So Son et al. adopted a pooling technique to embed nodes in the same sentence into one feature, and added this method to the work of other researchers, such as Wu et al.'s SiT, which improved the results to a certain extent. These techniques can examine the semantics of different AST structure levels and parts, focusing on processing node sets, and establish a certain degree of association with the information in the AST sequence and source code tokens. Due to the nature of AST as a graph, some researchers use graph neural networks to learn AST. MMTrans proposed by Yang et al. simultaneously applies graph convolutional networks (GCN) and transformers to learn AST from different angles through two styles of neural networks; Zhang et al. introduced a simple method combining graph attention networks (GAT) and transformers; Zhang et al. directly used GAT to encode AST sequences without special processing, and used Transformer to capture source code sequences.

[0005] To summarize the above, there are three major challenges in generating satisfactory code summaries: (1) Current works using pre-ordered traversal sequences of ASTs often miss the type of each node, resulting in insufficient learned features; (2) Many researchers enhance ASTs by adding data flow, control flow, or other edges, and design matrices to guide Transformers to learn these relationships; but they do not consider the importance of these edges and their surrounding context to the entire program interpretation; (3) How to capture program features of different granularities and levels and mix them in a more effective way. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a code summary generation method and device based on structural attention to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A code summary generation method based on structural attention includes the following steps:

[0009] S1. Parse the code snippet of the source code into an original abstract syntax tree, and split the nodes of the original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree;

[0010] S2. Based on the split abstract syntax tree, connect edges are constructed for the terminal value nodes, type nodes, and value nodes of the abstract syntax tree to obtain original edges.

[0011] Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges;

[0012] Build connecting edges for leaf nodes in the abstract syntax tree;

[0013] S3. Perform joint operations on the adjacency matrix of the original edge and the control flow, and on the adjacency matrix of the original edge and the data flow, respectively, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix, and calculate the quadratic power of the Origin-Control matrix and the Origin-Dataflow matrix respectively, and finally obtain two structure-based attention score matrices in which each element in the matrix represents the number of paths from one node to another node, and the number of paths is 2;

[0014] Construct an edge relationship matrix based on original edges, control flow edges, and data flow edges;

[0015] Perform a pre-order traversal of all nodes in the abstract syntax tree and create a one-hot matrix for all nodes in the abstract syntax tree;

[0016] S4. Construct a code summary model based on structural attention using a structure-based attention score matrix, a sequence of pre-order traversal, an edge relationship matrix, and a hierarchy matrix; the code summary model includes a GraphSAGE encoder, a Transformer encoder, and a fusion decoder;

[0017] The GraphSAGE encoder learns the features of local perspective in the abstract syntax tree sequence of pre-order traversal;

[0018] Based on the Transformer encoder, the embedding of each token in the code snippet is added to the embedding of the corresponding node in the abstract syntax tree sequence, and then concatenated with the hierarchical matrix;

[0019] The double cross attention mechanism deployed in the fusion decoder picks up the features of different perspectives of the abstract syntax tree sequence learned by the GraphSAGE encoder and the Transformer encoder based on the fusion decoder;

[0020] S5, generates code summaries using the constructed structured attention-based code summarization model.

[0021] Furthermore, the step S2 of obtaining the control flow edge according to the type node is specifically as follows:

[0022] When the type node is BlockStatement, connect the type nodes of its child nodes together;

[0023] When the type node is one of IfStatement, ForStatement, and WhileStatement, it is connected to the next sibling node and the type nodes of its child nodes are connected together.

[0024] Furthermore, the step S2 of obtaining the data flow edge according to the value node is specifically as follows:

[0025] Connect all value nodes from the defined location to the referenced location.

[0026] Further, the GraphSAGE encoder in step S4 includes: a transformer and an aggregator;

[0027] The transformer is used to adjust according to the attention score matrix of the structure;

[0028] The aggregator is used to capture relatively local features of each node in the abstract syntax tree sequence.

[0029] Furthermore, in step S4, the features of the local perspective in the abstract syntax tree sequence of the pre-order traversal are learned based on the GraphSAGE encoder, specifically:

[0030] BELU nonlinearity is introduced for each layer of nodes, and residual connections are used, where:

[0031] Node in layer k i The processing can be expressed as:

[0032]

[0033] in, Represents the features of the i-th node in the k-th layer; represents the feature of the i-th node in the k-1th layer; W1 and W2 represent learnable weight matrices; N(i) is the set of neighbors of the i-th node; Aggr is an aggregation function used to aggregate the information transmitted by the neighbors of the i-th node;

[0034] All features are connected in series and sent to the RELU activation function as shown below;

[0035]

[0036] in: is the total output of the entire GraphSAGE at layer k;

[0037] The residual connection and layer normalization strategies are used to generate local perspective features, specifically:

[0038]

[0039] Furthermore, the Transformer-based encoder adds the embedding of each token in the code snippet to the embedding of the corresponding node in the AST sequence, and then concatenates it with the hierarchical matrix, and is represented by the following steps:

[0040] emb ast =emb(s1,s2,...,s n )

[0041] emb code =emb(c1,c2,...,c m )

[0042]

[0043] otherwise;

[0044] h=[emb fusion ,M level ]

[0045] Where: emb ast Embedding vector representing the AST node sequence; emb code Embedding vector representing source code sequence; emb fusion represents the fusion embedding vector of the above two embedding vectors; h represents the final embedding vector combined with hierarchical information;

[0046] s i and c i denotes the i-th token in the abstract syntax tree sequence and the source code sequence respectively; Emb(·) is the embedding layer, M level is a level matrix.

[0047] The present invention also provides a device for generating a code summary based on structural attention, the device comprising:

[0048] A split processing module parses the code snippet of the source code into an original abstract syntax tree, and splits the nodes of the original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree;

[0049] S2. Based on the split abstract syntax tree, connect edges are constructed for the terminal value nodes, type nodes, and value nodes in the abstract syntax tree to obtain original edges.

[0050] Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges;

[0051] Build connecting edges for leaf nodes in the abstract syntax tree;

[0052] The module for generating the structural attention score matrix performs joint operations on the adjacency matrix of the original edge and the control flow, and on the adjacency matrix of the original edge and the data flow, respectively, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix, and finally obtains two structure-based attention score matrices in which each element in the matrix represents a path number of 2 from one node to another by calculating the quadratic power of the Origin-Control matrix and the Origin-Dataflow matrix respectively;

[0053] A model building module, which uses a structure-based attention score matrix, a sequence of pre-order traversal, an edge relationship matrix, and a hierarchy matrix to build a code summarization model based on structure attention: the code summarization model includes a GraphSAGE encoder, a Transformer encoder, and a fusion decoder;

[0054] The GraphSAGE encoder learns the features of local perspective in the abstract syntax tree sequence of pre-order traversal;

[0055] The Transformer encoder adds the embedding of each token in the code snippet to the embedding of the abstract syntax tree sequence, and then concatenates it with the hierarchical matrix;

[0056] The double cross attention mechanism deployed in the fusion decoder picks up the features of different perspectives of the abstract syntax tree sequence of the pre-order traversal captured by the GraphSAGE encoder and the Transformer encoder respectively;

[0057] Model output module: Generate code summary using the constructed structure-attention-based code summary model.

[0058] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0059] 1. The present invention analyzes the abstract syntax tree of the source code, splits each node according to the value and type field of each node, and reconstructs a new abstract syntax tree Split AST; then explores the data flow, control flow and its context information on the reconstructed new abstract syntax tree, constructs the corresponding attention score matrix, and then uses the GraphSAGE encoder and the Transformer encoder to extract multi-granularity and multi-representation features to enhance the effect of the code summary generation task. With respect to the traditional code summary generation method, the present invention makes the following improvements:

[0060] (1) Previously, researchers who often used source code AST to represent code usually only accessed the value of the node when traversing, but ignored the fact that the type of the node is explicit grammatical information. In the present invention, both the type and value of the AST node are taken into consideration, and the AST node is split into separate nodes in the task, and a new AST representation that explicitly contains grammatical information is reconstructed. This explicit structural representation allows the model to more easily capture the grammatical structural features of the source code.

[0061] (2) Existing methods usually focus on the added relationships themselves, but ignore that the context nodes around these edges may also enhance the model's understanding of the source code; and also ignore the importance of the involved relationships. In the present invention, by quadratic adjacency matrices and treating these matrices as attention score matrices instead of mask matrices, the code summarization model based on structural attention pays attention to the edges of the defined context nodes, rather than being limited to the original edges, data flow edges, and control flow edges themselves.

[0062] (3) Existing work using hybrid representation and granularity to generate code summaries requires a better way to capture and fuse their different features. In the present invention, the code summarization model based on structural attention includes two different basic models: GraphSAGE encoder and Transformer encoder. The former can learn AST with a higher degree of generalization, and the latter can capture global and comprehensive features in a more flexible way. The combination of the two enables the model to learn the features of the source code more comprehensively, thereby better generating code summaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] The code summary generation method and device based on structural attention of the present invention are further described below in conjunction with the accompanying drawings;

[0065] Figure 1 It is a flow chart of the code summary generation method based on structural attention provided by the present invention;

[0066] Figure 2 It is a schematic diagram of processing an abstract syntax tree in the code summary generation method based on structural attention provided by the present invention;

[0067] Figure 3 This is a framework diagram of the encoder-decoder model in the code summary generation method based on structural attention provided by the present invention. DETAILED DESCRIPTION

[0068] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0069] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, the present invention provides a code summary generation method based on structural attention, which specifically parses the code snippets of the source code into original abstract syntax trees, splits the abstract syntax trees, thereby obtaining split abstract syntax trees, and then performs a series of operations on the split abstract syntax trees to generate a structure-based attention score matrix;

[0071] Then, based on the generated structure-based attention score matrix, attention score matrix, abstract syntax tree sequence, relation matrix and hierarchy matrix are constructed respectively;

[0072] A code summary model based on structural attention is constructed based on the attention score matrix, abstract syntax tree sequence, relationship matrix and hierarchical matrix. Finally, the whole process of generating code summary using the constructed code summary model based on structural attention includes the following steps:

[0073] (1) The process of splitting the nodes of the abstract syntax tree into type nodes and value nodes:

[0074] The dataset is in the form of NL-PL pairs, where NL represents summary and PL represents code snippet. Figure 2As shown in , the code snippet is first parsed into a raw abstract syntax tree (AST), which is a tree structure that reflects the grammatical structure of the code; since most nodes have "type" and "value" fields, and only a few nodes have only the "type" field, the "type" field clearly indicates the grammatical role of the word in the source code, and the "value" field directly records the source code word itself. The "type" field is taken into account and the AST nodes are divided into type nodes and value nodes. The type node is regarded as the parent node of the value node, and the value node is split into more value nodes according to the child symbols in it, and the corresponding relationship is applied between the value nodes of each child symbol. The current state of the abstract syntax tree (AST) is shown in Figure 2 As shown in the SplitAST in , the black line represents the current edge as mentioned above, the green node represents the type node, and the blue node represents the value node.

[0075] (2) Perform preliminary processing on the split abstract syntax tree, construct the original edges, control flow edges, data flow edges, the adjacency matrix of the original edges, and the detailed process of splitting the edges between the leaving nodes of the split abstract syntax tree:

[0076] The construction process of the original edge: Figure 2 As shown by the blue arrows in the figure, since the terminal value nodes have sequential characteristics, all terminal value nodes are connected first. By traversing the terminal nodes in order, the sequence of the source code can be roughly restored. And these edges build many bridges for nodes to access more nodes, laying the foundation for subsequent steps. Figure 2 The blue arrow in the figure) and the edge mentioned in the previous section ( Figure 2 The edges between the black arrows in the figure are called primitive edges.

[0077] The construction process of control flow edges and data flow edges: explore the control flow and data flow relationships in the abstract syntax tree (AST) and add corresponding edges to express them. Figure 2 As shown by the red arrow in , for control flow, when a type node is BlockStatement, connect the type nodes of its child nodes together; if a type node is IfStatement, ForStatement, or WhileStatement, connect it to the next sibling node to comply with the real logic of the source code. Figure 2 As shown by the green arrows in , for data flow, all value nodes are connected from where they are defined to where they are referenced. The present invention also designs the adjacency matrix of the original edge, and the edges between the leaving nodes of the split AST, the data flow edges and the control flow edges.

[0078] (3) The process of performing a series of operations on the split abstract syntax tree to generate a structure-based attention score matrix: Although the above-mentioned relationships can relatively comprehensively represent the syntax and logic of the source code, previous researchers have only limited themselves to these relationships and have not considered the importance of the relationships between these nodes. However, other nodes adjacent to these nodes also contain some practical information. The context around the edges and corresponding nodes and the closeness of the connection obviously help program interpretation. In order to fill the gap of not considering the relationship between these nodes and other adjacent nodes, the present invention uses the quadratic power of the adjacency matrix, and each element in the resulting matrix represents the number of paths of length 2 from one node to another. Not only the local context of the node is taken into account, but also the closeness between the two-hop nodes is measured by the number of edges. The detailed process is: by performing joint operations on the adjacency matrix of the original edge and the control flow and data flow respectively, two matrices are obtained. Then, by calculating the quadratic power of these two matrices, the final two matrices are obtained. Since the original edge includes the edges between the leaving nodes, the edges between the leaving nodes can find bridges for the nodes to reach more other nodes. For example, it is noted that the control flow ( Figure 2 The red arrow in the figure) and the original edge ( Figure 2 Black and Figure 2 ), the direct children of the MethodDeclaration node can be related to each other because they are 2-hop neighbors; by considering the relationship between these nodes and other nodes, the model understands the general framework of the program. The number of paths from the FormalParameter node to the int node is greater than the number of paths to the Boolean node, indicating that the parameter syntax tends to use int rather than Boolean.

[0079] All of the above relationships are designed as adjacency matrices, i.e., structure-based attention score matrices, and all these edges are recorded as edge relationship matrices for use in GraphSAGE. In order to explicitly use the hierarchical position information in AST, the present invention also creates a one-hot matrix for all nodes as a hierarchical feature.

[0080] Specifically, S1. parse the code snippet of the source code into an original abstract syntax tree, and split the nodes of the original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree;

[0081] S2. Based on the split abstract syntax tree, connect edges are constructed for the terminal value nodes, type nodes, and value nodes of the abstract syntax tree to obtain original edges.

[0082] Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges;

[0083] Build connecting edges for leaf nodes in the abstract syntax tree;

[0084] S3. Perform joint operations on the adjacency matrix of the original edge and the control flow, and on the adjacency matrix of the original edge and the data flow, respectively, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix, and calculate the quadratic power of the Origin-Control matrix and the Origin-Dataflow matrix respectively, and finally obtain two structure-based attention score matrices in which each element in the matrix represents the number of paths from one node to another node, and the number of paths is 2;

[0085] Construct an edge relationship matrix based on original edges, control flow edges, and data flow edges;

[0086] Perform a pre-order traversal of all nodes in the abstract syntax tree and create a one-hot matrix for all nodes in the abstract syntax tree.

[0087] It should be noted that the AST sequence of pre-order traversal: starting from the root node, for the current node, we first read the content of this node, then traverse its child nodes one by one, and then read their contents, and repeat this process until all nodes have been traversed.

[0088] The level matrix of each node is a one-hot matrix:

[0089] The number of rows in the matrix is ​​the number of nodes, and the number of columns is the total number of layers in the AST. If the i-th node belongs to the j-th layer in the AST, then the value of the j-th element in the i-th row of this one-hot matrix is ​​1, and all other elements are 0.

[0090] After this stage, the structure-based attention score matrix, the sequence of pre-order traversal, the edge relationship matrix and the level matrix are obtained and input into the model of the next stage.

[0091] (4) The process of constructing a structure-based attention code summarization model using the structure-based attention score matrix, abstract syntax tree sequence, relation matrix, and hierarchy matrix is ​​as follows:

[0092] By applying the GraphSAGE encoder and the Transformer encoder (whose multi-head attention mechanism is adjusted) to learn different perspectives of the AST sequence, and deploying a double cross attention mechanism in the fusion decoder to generate the final summary. It includes: a split processing module that parses the code snippet of the source code into the original abstract syntax tree, and splits the nodes of the original abstract syntax tree into type nodes and value nodes to obtain the split abstract syntax tree;

[0093] S2. Based on the split abstract syntax tree, connect edges are constructed for the terminal value nodes, type nodes, and value nodes in the abstract syntax tree to obtain original edges.

[0094] Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges;

[0095] Build connecting edges for leaf nodes in the abstract syntax tree;

[0096] The module for generating the structural attention score matrix performs joint operations on the adjacency matrix of the original edge and the control flow, and on the adjacency matrix of the original edge and the data flow, respectively, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix, and finally obtains two structure-based attention score matrices in which each element in the matrix represents a path number of 2 from one node to another by calculating the quadratic power of the Origin-Control matrix and the Origin-Dataflow matrix respectively;

[0097] A model building module, which uses a structure-based attention score matrix, a sequence of pre-order traversal, an edge relationship matrix, and a hierarchy matrix to build a code summarization model based on structure attention: the code summarization model includes a GraphSAGE encoder, a Transformer encoder, and a fusion decoder;

[0098] The GraphSAGE encoder learns the features of local perspective in the abstract syntax tree sequence of pre-order traversal;

[0099] The Transformer encoder adds the embedding of each token in the code snippet to the embedding of the abstract syntax tree sequence, and then concatenates it with the hierarchical matrix;

[0100] The double cross attention mechanism deployed in the fusion decoder picks up features from different perspectives of the abstract syntax tree sequence of the pre-order traversal captured by the GraphSAGE encoder and the Transformer encoder respectively;

[0101] Model output module: Generate code summaries using the constructed structure-attention-based code summarization model; the details are as follows:

[0102] ①GraphSAGE encoder: The obtained AST sequence of the pre-order traversal will be input into the code summarization model based on structural attention. Each token represents each node in SplitAST. The GraphSAGE encoder in the code summarization model based on structural attention consists of two parts: a transformer and an aggregator; among them, the transformer is adjusted to utilize the structure-based attention score matrix. And the GraphSAGE encoder is used to capture relatively local features. GCN needs to input the entire graph when calculating node features, while GraphSAGE only inputs the subgraph, which obviously reduces the complexity of the calculation. In addition, the GraphSAGE encoder learns an aggregator rather than a specific node embedding, so the GraphSAGE encoder has stronger generalization ability. RELU nonlinearity is used in each layer, and residual connections are used. Node in the kth layer i The processing can be expressed as

[0103]

[0104] in, represents the feature of the i-th node in the k-1th layer; W1 and W2 represent learnable weight matrices; N(i) is the set of neighbors of the i-th node; Aggr is an aggregation function used to aggregate the information transmitted by the neighbors of the i-th node. Then, all the features are connected in series and sent to the RELU activation function.

[0105]

[0106] Residual connections and layer normalization strategies are used to overcome the gradient vanishing and excessive offset problems caused by multi-layer processing.

[0107]

[0108] This generates local-perspective features that facilitate the model to learn the entire program in a microscopic manner.

[0109] ②Transformer encoder: First, the input AST sequence embedding is processed in two steps, combining the sequence and level information with the AST sequence.

[0110] The first step of embedding is source code token fusion. Code snippets explicitly describe the order information. Although the edges of terminal nodes are considered in the structure-based attention score matrix, this information is explicitly fused by adding the embedding of each token in the code snippet to the embedding of the corresponding node in the AST sequence. This is because the code token embedding contains order and position encoding information. The second step is to supplement the hierarchical features. The level of each node in the AST is the basic information of the node feature. In order to clearly show this information, this hierarchical matrix is ​​concatenated with the embedding of the AST sequence. These two steps can be expressed as

[0111] emb ast =emb(s1,s2,...,s n )

[0112] emb code =emb(c1,c2,...,c m )

[0113]

[0114] otherwise;

[0115] h=[emb fusion ,M level ]

[0116] where s i and c i represents the i-th token in the AST sequence and the source code sequence respectively. Emb(·) is the embedding layer, M level is a hierarchical matrix. By fusing sequence and hierarchical information, AST sequence embedding will bring rich representation to the learning of Transformer encoder.

[0117] The code snippet contains a lot of logic and different processes, which need to be grasped in a more comprehensive way. The features captured by GraphSAGE or other graph neural networks are often relatively local. Therefore, an adjusted Transformer encoder is used to extract these features. Gao et al. believe that in attention calculations, the features and relationships extracted by different attention heads are redundant. Therefore, Gao et al. apply matrices to different heads in different layers. The process can be expressed as

[0118]

[0119] Among them, Q, K, and V are the projections of query, key, and value respectively. is a scaling factor that mitigates the effect of larger values. M r ∈R n×nis the attention score matrix, n is the number of nodes, and the implementation method of the present invention is different from that of Gao et al. in that: first, Gao et al. regard these matrices as mask matrices, setting the elements irrelevant to the relationship to -∞, thereby emphasizing the elements in the relationship. Nevertheless, other elements should not be discarded, all elements are important to varying degrees, and over-emphasizing a few specific nodes may weaken the overall information. The present invention increases the scores between related nodes by several times instead of reducing the attention scores of irrelevant nodes to negative infinity. Secondly: Gao et al. specify a fixed number of heads for various types of matrices, while the present invention sets learnable parameters for different structure-based attention score matrices, which more flexibly guides the model to grasp information from different angles and to varying degrees.

[0120] ③ Fusion decoder: such as Figure 3 As shown, the present invention uses two cross-attention modules in the decoder layer of Transformer, which pick up the features of different perspectives captured by GraphSAGE and Transformer encoder respectively, and input them into their respective cross-attention as Q and K. Then, the output of the fusion decoder will be mapped to the probability distribution of vocabulary.

[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A code summary generation method based on structural attention, characterized in that: The following steps are involved: S1. Parse the code snippet of the source code into an original abstract syntax tree, and split the nodes of the original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree; S2. Based on the split abstract syntax tree, connect edges are constructed for the terminal value nodes, type nodes, and value nodes in the abstract syntax tree to obtain original edges. Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges; Build connecting edges for leaf nodes in the abstract syntax tree; S3. Perform joint operations on the adjacency matrix of the original edge and the control flow, and on the adjacency matrix of the original edge and the data flow, respectively, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix, and calculate the quadratic power of the Origin-Control matrix and the Origin-Dataflow matrix respectively, and finally obtain two structure-based attention score matrices in which each element in the matrix represents the number of paths from one node to another node, and the number of paths is 2; Construct an edge relationship matrix based on original edges, control flow edges, and data flow edges; Perform a pre-order traversal of all nodes in the abstract syntax tree and create a one-hot matrix for all nodes in the abstract syntax tree; S4. Constructing a code summary model based on structural attention using a structure-based attention score matrix, a pre-order traversal sequence, an edge relationship matrix, and a hierarchy matrix; the code summary model includes a GraphSAGE encoder, a Transformer encoder, and a fusion decoder; The GraphSAGE encoder learns the features of local perspective in the abstract syntax tree sequence of pre-order traversal; Based on the Transformer encoder, the embedding of each token in the code snippet is added to the embedding of the corresponding node in the abstract syntax tree sequence, and then concatenated with the hierarchical matrix; A double cross attention mechanism is deployed in the fusion decoder, and the features of different perspectives of the abstract syntax tree sequence of the pre-order traversal learned by the fusion decoder are picked up respectively by the GraphSAGE encoder and the Transformer encoder; S5. Generate code summaries using the constructed structure-attention-based code summarization model.

2. The code summary generation method based on structural attention according to claim 1 is characterized in that: S2 obtains the control flow edge according to the type node, specifically: When the type node is BlockStatement, connect the type nodes of its child nodes together; When the type node is one of IfStatement, ForStatement, and WhileStatement, it is connected to the next sibling node and the type nodes of its child nodes are connected together.

3. The code summary generation method based on structural attention according to claim 1 is characterized in that: S2 obtains data flow edges according to value nodes, specifically: Connect all value nodes from where they are defined to where they are referenced.

4. The code summary generation method based on structural attention according to claim 1 is characterized in that: The S4, GraphSAGE encoder, includes: a transformer and an aggregator; The transformer is used to adjust according to the attention score matrix of the structure; The aggregator is used to capture the local features of each node in the abstract syntax tree sequence.

5. The code summary generation method based on structural attention according to claim 1 is characterized in that: S4, based on the GraphSAGE encoder, learns the features of the local perspective in the abstract syntax tree sequence of the pre-order traversal, specifically: BELU nonlinearity is introduced for each layer of nodes, and residual connections are used, where: Node in layer k i The processing statement is: in, Represents the features of the i-th node in the k-th layer; represents the feature of the i-th node in the k-1th layer; W1 and W2 represent learnable weight matrices; N(i) is the set of neighbors of the i-th node; Aggr is an aggregation function used to aggregate the information transmitted by the neighbors of the i-th node; All features are connected in series and sent to the RELU activation function as shown below; in: is the total output of the entire GraphSAGE at layer k; The residual connection and layer normalization strategies are used to generate local perspective features, specifically:

6. The code summary generation method based on structural attention according to claim 1 is characterized in that: The Transformer-based encoder adds the embedding of each token in the code snippet to the embedding of the corresponding node in the AST sequence, and then concatenates it with the hierarchical matrix and represents it through the following steps: emb ast =emb(s1,s2,...,s n ) emb code =emb(c1,c2,...,c m ) h=[emb fusion ,M level ] Where: emb ast Embedding vector representing the AST node sequence; emb code Embedding vector representing source code sequence; emb fusion represents the fusion embedding vector of the above two embedding vectors; h represents the final embedding vector combined with hierarchical information; s i represents the i-th token in the abstract syntax tree sequence; c j represents the jth word in the source code sequence; emb(·) is the embedding layer; M level is a level matrix.

7. A device for generating code summaries based on structural attention, characterized in that: The device is applied to the code summary generation method based on structural attention according to any one of claims 1 to 6, and the device comprises: A split processing module parses the code snippet of the source code into an original abstract syntax tree, and splits the nodes of the original abstract syntax tree into type nodes and value nodes to obtain a split abstract syntax tree; Based on the split abstract syntax tree, connecting edges are constructed for the terminal value nodes, type nodes, and value nodes in the abstract syntax tree to obtain original edges; Based on the control flow and data flow in the abstract syntax tree, construct control flow edges and data flow edges, and design the adjacency matrix of the original edges; Build connecting edges for leaf nodes in the abstract syntax tree; The generation module performs joint operations on the adjacency matrix of the original edge and the control flow, as well as the adjacency matrix of the original edge and the data flow, to preliminarily obtain the Origin-Control matrix and the Origin-Dataflow matrix. By calculating the second power of the Origin-Control matrix and the Origin-Dataflow matrix respectively, we finally obtain two structure-based attention score matrices in which each element in the matrix represents the number of paths from one node to another node, which is 2. Model building module, which builds edge relationship matrix based on original edges, control flow edges and data flow edges; Perform a pre-order traversal of all nodes in the abstract syntax tree and create a one-hot matrix for all nodes in the abstract syntax tree; A code summarization model based on structural attention is constructed by using a structure-based attention score matrix, a sequence of pre-order traversal, an edge relationship matrix and a hierarchical matrix; the code summarization model includes a GraphSAGE encoder, a Transformer encoder and a fusion decoder; The GraphSAGE encoder learns the features of local perspective in the abstract syntax tree sequence of pre-order traversal; Based on the Transformer encoder, the embedding of each token in the code snippet is added to the embedding of the corresponding node in the abstract syntax tree sequence, and then concatenated with the hierarchical matrix; A double cross attention mechanism is deployed in the fusion decoder, and the features of different perspectives of the abstract syntax tree sequence of the pre-order traversal learned by the fusion decoder are picked up respectively by the GraphSAGE encoder and the Transformer encoder; Model output module: Generate code summary using the constructed structure-attention-based code summary model.

Images