A code summarization method for graph sequence association based on attention mechanism

Abstract

This invention relates to the field of code summarization technology, specifically to a code summarization method based on attention mechanism and graph sequence association, comprising: 1. Processing the code: segmenting the source code into words to obtain a code sequence; parsing the code to obtain an abstract syntax tree; and obtaining a code graph by adding a sequence stream and a data stream; 2. Encoding the code sequence and code graph using a sequence encoder and a graph encoder respectively, to obtain a sequence encoding vector and a graph encoding vector; 3. At the decoder end, inputting the starting word of the code summarization. <bos>In the decoder, attention is calculated simultaneously based on both the sequence encoding vector and the graph encoding vector. After fusion, the decoder output, i.e., the predicted word, is obtained through a fully connected mapping. Fourth, the output word of the previous time step is used as the input of the decoder at the current time step, and step three is repeated until the decoder output encounters the terminating word. <eos>End. This invention provides a superior code summary.< / eos> < / bos>

Description

Technical Field

[0001] This invention relates to the field of code summarization technology, and more specifically, to a code summarization method based on attention mechanism for graph sequence association. Background Technology

[0002] Code summarization is a task that describes code using natural language, including descriptions of the code's function, purpose, and considerations—all beneficial for program understanding. As software codebases grow increasingly large, nearly 90% of development time is spent on software maintenance, such as version iteration, program understanding, and bug fixing. This extended period of program understanding undoubtedly has a significant impact on software development efficiency. In fact, most code comments suffer from problems such as missing, insufficient, or mismatched comments due to human error. The level of importance developers place on comments often determines whether they can be effectively understood. High-quality code summaries can greatly reduce the overhead of developers in program understanding. By providing concise code descriptions, code summaries allow developers to quickly grasp the code's function and usage. Therefore, code summary generation is a very meaningful task for developers; high-quality code comments can significantly reduce unnecessary costs during the development process.

[0003] Current research on code summarization largely relies on retrieval-based or generation-based methods. A major problem with retrieval-based code summarization is its poor performance when encountering code that differs significantly from the database. Generative models typically take three inputs: code sequences, AST trees, and graphs. However, these models often obtain code summaries through sequences or flattened AST trees, failing to effectively consider the structural information of the code. Summary of the Invention

[0004] The present invention provides a code summarization method for graph sequence association based on an attention mechanism, which can overcome some or all of the defects of the prior art.

[0005] The code summarization method for graph sequence association based on attention mechanism according to the present invention is characterized by comprising the following steps:

[0006] First, the code is processed by segmenting the source code into a code sequence; the code is parsed using the open-source tool tree-sitter to obtain an abstract syntax tree; and a code graph is obtained by adding sequence streams and data streams.

[0007] 2. The code sequence and code graph are encoded by a sequence encoder and a graph encoder respectively to obtain the sequence encoding vector and the graph encoding vector;

[0008] 3. On the decoder side, input the starting word of the code summary.<bos>In the decoder, attention is calculated simultaneously based on both the sequence encoding vector and the graph encoding vector. After fusion, the decoder output, i.e. the predicted word, is obtained through a fully connected mapping.

[0009] Fourth, use the output word from the previous time step as the input to the decoder at the current time step, and repeat step three until the decoder outputs the terminating word. <eos>Finish.

[0010] As a preferred choice, the sequence encoder is a Transformer, which employs a multi-head self-attention mechanism, as detailed below:

[0011] Given an input sequence X = {x1, x2, x3, ..., x...} n },in R is a set of vector spaces, d model This represents the dimension of the vector; the calculated output is Output = {o1, o2, o3, ..., o n First, initialize 4 d values. model ×d model Learnable matrix W Q W K W V W O The output of multi-head attention is calculated using the following formula:

[0012] Output=Concat(head1,...,head h W O

[0013]

[0014] Q i ,K i V i =XW i Q XW i K XW i V

[0015] Among them W i Q W i K , and d k =d model / / h; h is the number of heads; the final output is obtained by concatenating the outputs of multiple heads and then mapping them.

[0016] As a preferred embodiment, the graph encoder is a graph neural network (GAT), as follows:

[0017] Given an input graph G = (V, E, X), where V, E, and X represent the set of nodes, the set of edges, and the set of node vectors, respectively; for any e ∈ E, e ∈ R D×D D is the dimension of the adjacency matrix; GAT is implemented by stacking simple graph attention layers; graph attention calculates the attention coefficient for node pairs (i,j) as follows:

[0018]

[0019] Where α ij N is the attention coefficient from node j to node i. i Let n represent the set of neighboring nodes of node i; the input features of a node are n = {n1, n2, ..., nn}. N }, and h i ∈R D ,h i Here, N is the hidden layer vector of node i, and N and D represent the number of nodes and the vector dimension, respectively; the output feature of the node is O = {o1,...,o...} N }, o∈R D′ ;W∈R D′×D It is the linear transformation weight matrix applied at each node, where D' is the dimension of the transformation matrix and a is the weight vector; LeakyReLU is a non-linear activation function, normalized using softmax;

[0020] The final output of the node is obtained according to the following formula:

[0021]

[0022] Where W is the linear transformation matrix; σ is the nonlinear activation function.

[0023] Preferably, in step three, the output G = {g1,...,g...} of the graph encoder is... n The outputs of the sequence encoder S = {s1,...,s} n }; Calculate the score according to the following formula, and use the score to obtain the correlation between sequence information and structural information:

[0024]

[0025] score i,j =softmax(a i,j )

[0026] in W is a learnable matrix.

[0027] This invention proposes a static code graph (SCG) based on an abstract syntax tree (AST). The graph adds data flow and sequence flow to the AST. These two types of information are encoded separately by a sequence encoder and a graph encoder, respectively, and an attention mechanism is used to associate the sequence-encoded information and the graph-encoded information. This invention comprehensively considers both sequence and structural information, and during decoding, it considers the feature encoding that has the greatest impact on the generated words, thus achieving better code summarization. Attached Figure Description

[0028] Figure 1 This is a flowchart of the code summarization method for graph sequence association based on the attention mechanism in Example 1;

[0029] Figure 2 This is a schematic diagram of the calculation method for attention score in Example 1. Detailed Implementation

[0030] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0031] Example 1

[0032] like Figure 1 As shown, this embodiment proposes a sequence-graph combined code summarization model (TGCS). This model, based on the Transformer, combines graph neural networks to encode sequences and graphs respectively. TGCS includes the following steps:

[0033] First, the code is processed by segmenting the source code into a code sequence; the code is parsed using the open-source tool tree-sitter to obtain an abstract syntax tree; and a code graph is obtained by adding sequence streams and data streams.

[0034] 2. The code sequence and code graph are encoded by a sequence encoder and a graph encoder respectively to obtain the sequence encoding vector and the graph encoding vector;

[0035] 3. On the decoder side, input the starting word of the code summary. <bos>In the decoder, attention is calculated simultaneously based on both the sequence encoding vector and the graph encoding vector. After fusion, the decoder output, i.e. the predicted word, is obtained through a fully connected mapping.

[0036] Fourth, use the output word from the previous time step as the input to the decoder at the current time step, and repeat step three until the decoder outputs the terminating word. <eos>Finish.

[0037] The sequence encoder is a Transformer, which employs a multi-head self-attention mechanism to effectively capture long dependencies. Furthermore, the multi-head approach allows the model to learn relevant information representations in different subspaces. Specifically:

[0038] Given an input sequence X = {x1, x2, x3, ..., x...} n },in R is a set of vector spaces, d model This represents the dimension of the vector; the calculated output is Output = {o1, o2, o3, ..., o n First, initialize 4 d values. model ×d model Learnable matrix W Q W K W V W O The output of multi-head attention is calculated using the following formula:

[0039] Output=Concat(head1,...,head h W O

[0040]

[0041] Q i ,K i V i =XW i Q XW i K XW i V

[0042] Among them W i Q W i K , and d k =d model / / h; h is the number of heads; the final output is obtained by concatenating the outputs of multiple heads and then mapping them.

[0043] Traditional GCNs cannot be directly applied to directed graphs, while the SCG proposed here is a type of directed graph. To better encode structural information, a GAT (Graph Attention Network) is used to encode the SCG after node embedding. GAT employs a self-attention mechanism, where the hidden state of a node is obtained by calculating attention scores over its neighborhood. The graph encoder is the graph neural network GAT, as detailed below:

[0044] Given an input graph G = (V, E, X), where V, E, and X represent the set of nodes, the set of edges, and the set of node vectors, respectively; for any e ∈ E, e ∈ R D×D D is the dimension of the adjacency matrix; GAT is implemented by stacking simple graph attention layers; graph attention calculates the attention coefficient for node pairs (i,j) as follows:

[0045]

[0046] Where α ij N is the attention coefficient from node j to node i. i Let n represent the set of neighboring nodes of node i; the input features of a node are n = {n1, n2, ..., nn}. N }, and h i ∈R D ,h i Here, N is the hidden layer vector of node i, and N and D represent the number of nodes and the vector dimension, respectively; the output feature of the node is O = {o1,...,o...} N }, o∈R D′ ;W∈R D′×D It is the linear transformation weight matrix applied at each node, where D' is the dimension of the transformation matrix and a is the weight vector; LeakyReLU is a non-linear activation function, normalized using softmax;

[0047] The final output of the node is obtained according to the following formula:

[0048]

[0049] Where W is the linear transformation matrix; σ is the nonlinear activation function.

[0050] In step three, encoded vectors for two types of information are obtained from the sequence encoder and graph encoder. At this point, the two types of encoded information are unrelated. SCG is a graph obtained based on code parsing transformation, so there is a certain correlation between the nodes of the graph and the tokens of the sequence; that is, the sequence tokens also exist in the set of nodes of the graph. Since a graph is structural information, it does not have a sequential relationship. Therefore, in order to associate the sequence information and the structural information, an attention mechanism is used to associate the encoded vector of each node with the encoded vector of the sequence using scores. This forms a correlation between the structural information and the sequence information. Figure 2 The calculation form of the attention score is shown. For the output G = {g1,...,g...} of the graph encoder... n The outputs of the sequence encoder S = {s1,...,s} n }; Calculate the score according to the following formula, and use the score to obtain the correlation between sequence information and structural information:

[0051]

[0052] score i,j =softmax(a i,j )

[0053] in W is a learnable matrix.

[0054] The decoder uses a standard transformer decoder. However, to consider the impact of structural information on decoding, the output of the graph encoder is also fed into the decoder. The sequence information and structural information after multi-head attention are added together to obtain the final multi-head attention output.

[0055] The copy mechanism is an effective way to solve the OOV (Out of Context) problem at the output stage. It is used during word generation if an OOV is encountered... <unk>The word is then selected from the source code input to replace the selected token. <unk>It is possible to obtain a product that does not contain <unk>The summary is generated by using an additional attention layer in the last layer of the decoder to calculate the copy score. This mechanism allows the model to copy rare words to the generator, improving the summary performance.

[0056] The experimental design and methods will be discussed below, along with the datasets, baselines, and evaluation metrics used in the experiment. The relevant results will then be analyzed.

[0057] The models were trained and tested using Java and Python datasets. Table 1 shows the statistics for the two datasets. Tree parsing tools were used to parse the Java and Python code into AST trees, converting them into a graph structure representation of the code by adding data streams and sequence streams. To reduce the impact of out-of-vocabulary (OOV) errors, sub-tokens were used to segment concatenated words in the source code. Table 2 shows the change in code vocabulary size before and after using sub-tokens.

[0058] Table 1 Statistics of the Java dataset

[0059]

[0060] Table 2 shows the change in code vocabulary size before and after using sub-tokens.

[0061]

[0062]

[0063] In the experiments, three widely used evaluation metrics for code summarization—BLEU, METEOR, and ROUG—were used to evaluate the performance of the model.

[0064] The principle of BLEU is to calculate the number of n-gram matches between the generated sequence and the reference sequence and then calculate the average, where n = 1, 2, ..., N. The calculation formula is as follows:

[0065]

[0066]

[0067] in N represents the number of cn-gram matches between the generated sequence and the reference sequence. n ) represents the total number of reference sequence n-grams, and ρ is the brevity penalty.

[0068] ROUGE is a recall-based evaluation metric. In this experiment, ROUGE-LCS (Longest Common Sequence) was used as the ROUGE evaluation metric. ROUGE-LCS is calculated based on the longest common subsequence between the generated sequence and the reference sequence. The calculation formula is shown below:

[0069]

[0070]

[0071]

[0072] Where LCS(g,c) represents the longest common subsequence between the generated sequence g and the reference sequence c. The magnitude of β determines the degree of attention given to Recall and Precision.

[0073] METEOR was proposed to address some inherent shortcomings of BLEU. It uses WordNet to calculate specific sequence matches, synonyms, roots and affixes, and matching relationships between definitions, making it more relevant to human judgment.

[0074] This embodiment follows the hyperparameter settings of Ahmad et al. The maximum lengths of the code and summary are set to 150 and 50 respectively, with maximum dictionary sizes of 50,000 and 30,000 respectively. To demonstrate the complete structure of the code graph, the graph is not pruned; the complete graph structure is preserved. The dictionary size of the graph nodes is set to 50,000, the same as that of the code. This embodiment uses the Adam optimizer with an initial learning rate of 0.0001 to train the model. The model hyperparameters in the transformer are set according to the parameters in [reference needed]. Due to hardware limitations, the batch size is set to 16, and dropout is set to 0.2. The maximum training epochs are set to 200, and an early stopping mechanism is used. If the performance on the validation set does not improve for 20 consecutive epochs, training will stop to prevent overfitting. During the inference stage, the beamsearch algorithm with a beam size of 4 is used to generate sequences.

[0075] To demonstrate the superiority of this model over other models, this example is compared with the following baselines model:

[0076] CODE-NN employs an LSTM-based Sequence-to-Sequence architecture with an attention mechanism, using code snippets as input to generate corresponding code summaries.

[0077] Tree2Seq is a machine translation model that generates sequences based on tree structures. It encodes and decodes trees based on Tree-LSTM.

[0078] RL+Hybrid2seq uses an actor-critic network to obtain hidden layer vectors, and uses BLEU as a reward to guide the model's learning.

[0079] DeepCom uses SBT as the structural information of the code as input to assist in obtaining sequence information for the code. It also proposes a hybrid attention module that fuses different features.

[0080] API+CODE considers both code sequences and the API relationships within them in the code summarization task, using two different encoder-decoder models to encode and decode the API sequences and code sequences respectively. The learned API encoding is then used for code summarization decoding.

[0081] The Dual Model employs dual learning, leveraging the complementary relationship between code summarization and code generation tasks to learn from each other and thus improve the performance of code summarization.

[0082] The Transformer was used for the first time for code digest tasks, and the copy mechanism and relative position encoding were also added.

[0083] Code-Bert uses BERT for code summarization tasks, while flattening the data stream into a sequence and feeding it into BERT to enhance the code encoding information.

[0084] Table 3 presents the comparison results between the experiment and other experiments. Through the comparison, it can be seen that on the same dataset, the method of this embodiment shows corresponding improvements in BLEU, METEOR, and ROUGE metrics compared to baseline models. Compared with the Transformer baseline, improvements of 0.38 and 0.33 in BLEU and METEOR respectively demonstrate the improvement of the method in this embodiment. Since the BERT model performs better with large datasets, the small dataset in this embodiment cannot fully leverage the advantages of BERT. Compared with other baselines, the performance of the method in this embodiment on the same dataset is clearly visible. This embodiment also conducted related ablation experiments on the model. In Table 3, we compared the fusion methods of the decoded vectors of the two types of information in the decoder stage. `mul` indicates that the fusion method of the two decoded vectors is bitwise multiplication, and `add` indicates that the two decoded vectors are added. It can be seen that addition yields a better decoded vector than multiplication. This may be because the two encoded vectors are in two relatively independent vector spaces. When using dot multiplication, the two become blurred, resulting in a more chaotic encoded vector, making the model's performance even worse than that of baseline models.

[0085] Table 3 Comparison Results

[0086]

[0087] Meanwhile, we considered the impact of different GNNs on our graph encoding. Table 4 shows the effect of using two different GNNs on our model. In our experiments, we used GGNN (Gated Graph Neural Network) and GAT as the graph encoder of our model, respectively. The results show that using GAT, a self-attention-based graph convolutional neural network, achieves better performance than graph recurrent neural networks. Our analysis is that for large graphs, graph recurrent neural networks encode node vectors by calculating from neighboring nodes, while GAT, based on convolution and self-attention mechanisms, can obtain the encoded vector of each node globally.

[0088] Table 4. Model performance on different graph encoder networks.

[0089]

[0090] In the decoder, we also considered the impact of using a single multi-head attention module for decoding versus two multi-head attention modules on model performance, as shown in Table 5. `single` represents one multi-head attention block, and `bi` represents two multi-head attention blocks. For using a single multi-head attention module in the decoding stage, we concatenate the outputs of the sequence encoder and the graph encoder, assuming the sequence encoding vector S∈R. m×d ,Graph encoding vector G∈R n×d Where m represents the sequence length, n represents the number of nodes, d represents the hiddendim of the encoded vector, and our decoder's input vector C∈R (m+n)×d Using two multi-head attention modules, we input the two encoded vectors into different modules and sum their outputs to obtain the final output. Experiments show that using a single multi-head attention module performs significantly worse than using two, and even worse than some baseline models. Analysis reveals that the number of graph nodes in the decoder input vector obtained through concatenation is generally much greater than the sequence length. Because we set the maximum sequence length to 150 but did not limit the number of graph nodes, structural information dominates the concatenated vector, diluting the sequence information and resulting in poor model performance.

[0091] Table 5 Model performance on different numbers of multi-head attention blocks.

[0092]

[0093] In the inference stage, we use beam search to generate the summary. Beam search is an improved algorithm of greedy search. At each time step, instead of selecting the highest-scoring option, we retain all possible beam sizes, obtaining the possible beam sizes at each time step, sacrificing time for a better solution. Table 6 shows the impact of different beam sizes on the experimental results, where beam size = 1 indicates the use of greedy decoding. When we use greedy decoding, we have achieved the same BLEU score as the transformer using beam search, and improved the METEOR score.

[0094] Table 6 Performance of different beam sizes

[0095]

[0096]

[0097] This embodiment proposes a model, TGCS, that comprehensively considers both sequence and structural information in code. It effectively captures both information and fuses them through attention. Experiments demonstrate that our method improves upon previous research on relevant evaluation metrics.

[0098] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.< / unk> < / unk> < / unk> < / eos> < / bos> < / eos> < / bos>

Claims

1. A code summarization method for graph sequence association based on attention mechanism, characterized in that: Includes the following steps: First, the code is processed by segmenting the source code into a code sequence; the code is parsed using the open-source tool tree-sitter to obtain an abstract syntax tree; and a code graph is obtained by adding sequence streams and data streams.

2. The code sequence and code graph are encoded by a sequence encoder and a graph encoder respectively to obtain the sequence encoding vector and the graph encoding vector; 3. On the decoder side, input the starting word of the code summary. <bos> In the decoder, attention is calculated simultaneously based on both the sequence encoding vector and the graph encoding vector. After fusion, the decoder output, i.e. the predicted word, is obtained through a fully connected mapping.< / bos> Fourth, use the output word from the previous time step as the input to the decoder at the current time step, and repeat step three until the decoder outputs the terminating word. <eos> Finish;< / eos> The sequence encoder is a Transformer, which employs a multi-head self-attention mechanism, as detailed below: For a given input sequence ,in , R It is a set of vector spaces. d model Represents the dimension of the vector; the calculated output First, initialize 4 Learnable matrix The output of multi-head attention is calculated using the following formula: ； in ,and h represents the number of heads; the final output is obtained by concatenating the outputs of multiple heads and then mapping them. The graph encoder is a graph neural network (GAT), as detailed below: Given input graph ,in Let them represent the set of nodes, the set of edges, and the set of node vectors, respectively; for D is the dimension of the adjacency matrix; GAT is implemented by stacking simple graph attention layers; graph attention targets node pairs. The attention coefficient is calculated as follows: ； in It is a node To the node Attention coefficient Represents a node The set of neighboring nodes; the input features of the nodes are ,and , node i The hidden layer vector, and These represent the number of nodes and the vector dimension, respectively; the output features of the nodes. , ; It is the linear transformation weight matrix applied at each node, where D' is the dimension of the transformation matrix and a is the weight vector; It is a non-linear activation function, using Perform normalization; The final output of the node is obtained according to the following formula: ； in It is a linear transformation matrix; It is a non-linear activation function.

2. The code digest method for graph sequence association based on attention mechanism according to claim 1, characterized in that: In step three, the output of the graph encoder... and the output of the sequence encoder Calculate the score using the following formula, and use the score to obtain the correlation between sequence information and structural information: ； in , It is a learnable matrix.

Images