Code completion method based on grouping attention

Abstract

The invention relates to the technical field of artificial intelligence and software engineering, in particular to a code completion method based on grouping attention, and the method comprises the steps: obtaining to-be-completed code data; the to-be-complemented code data is input into a preset code complementing model, complemented code data is output, the code complementing model is obtained based on training of a training set, the training set comprises a plurality of open source code data, and the code complementing model is used for constructing the input code data into input representation and outputting the input representation. And processing through a grouping attention mechanism and a feedforward neural network to complete code completion prediction output. According to the method, the model calculation cost can be reduced, and meanwhile, the code completion performance is kept or improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and software engineering, and particularly relates to a code completion method based on grouped attention. BACKGROUND

[0002] In modern software development processes, code completion tools have become an indispensable auxiliary means, which can significantly improve development efficiency and reduce errors. Early code completion methods usually rely on rule-based, template matching or simple statistical language models. With the development of machine learning and deep learning technologies, researchers began to apply these technologies to code completion tasks. The ability of recurrent neural networks (RNN) and its variants (such as LSTM and GRU) in sequence modeling makes them be used to capture the context information of code and improve the performance of code completion to some extent.

[0003] However, RNN-based models face the problem of gradient vanishing / explosion when dealing with long sequences, and their inherent sequential computation characteristics limit parallel processing capabilities, resulting in low processing efficiency. In recent years, models based on the Transformer architecture have achieved remarkable results in code completion tasks due to their strong parallel computing capabilities and the ability to capture long-distance dependencies through self-attention mechanisms. Many advanced code completion models have adopted the Transformer structure.

[0004] Although Transformer-based code completion models have achieved success, the core multi-head self-attention mechanism has a significant efficiency problem: the computational complexity grows quadratically with the length of the input sequence. This means that when dealing with long code files or needing to consider context, the model's computational resource requirements will increase dramatically, leading to slower code completion and affecting user experience. Therefore, how to effectively reduce the computational complexity of the model and improve efficiency while maintaining or even improving the performance of code completion is a key problem that needs to be solved in the current field. SUMMARY

[0005] The purpose of the present application is to solve the technical problem of high computational complexity of the multi-head self-attention mechanism in existing Transformer architecture-based code completion models, especially the low efficiency when dealing with long sequences, which affects the development experience. A code completion method based on grouped attention is provided to reduce the computational cost of the model while maintaining or improving the performance of code completion.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] A code completion method based on grouped attention, comprising:

[0008] obtaining code data to be completed;

[0009] inputting the to-be-completed code data into a preset code completion model, and outputting completed code data, wherein the code completion model is obtained based on a training set, the training set includes a plurality of open source code data, and the code completion model is used to construct input code data into input representation, process the input representation through a grouping attention mechanism and a feedforward neural network, and complete code completion prediction output.

[0010] Optionally, the code completion model is further preprocessed before being trained based on the open source code data in the training set, and the preprocessing includes:

[0011] parsing the open source code data to generate a corresponding abstract syntax tree;

[0012] splitting the abstract syntax tree into a plurality of sub-trees, identifying leaf nodes and non-leaf nodes from the sub-trees, and determining a root path corresponding to each leaf node;

[0013] performing frequency statistics on all nodes of the abstract syntax tree, and selecting a node with the highest frequency to construct a vocabulary table, wherein the vocabulary table contains special markers for representing unknown nodes and padding;

[0014] vocabulary of the leaf node sequence and the non-leaf node sequence in the corresponding root path according to the vocabulary table, and converting a digital index sequence.

[0015] Optionally, the code completion model includes:

[0016] an input representation construction layer for fusing a leaf node sequence of input code data and corresponding root path information to construct input representation;

[0017] a grouping attention calculation layer for applying a grouping attention mechanism to the input representation, calculating attention scores between leaf nodes, and obtaining attention output containing context information;

[0018] a feedforward network processing layer for inputting the grouping attention output into a feedforward neural network to learn and represent complex nonlinear relationships;

[0019] a prediction output layer for mapping the output of the feedforward neural network to the size of the vocabulary table through a linear layer, calculating the probability of each word in the vocabulary table as a target node, and outputting a prediction result of code completion.

[0020] Optionally, the input representation construction layer fuses the leaf node sequence of the input code data and the corresponding root path information to construct the input representation, and the construction includes:

[0021] indexing the leaf nodes in the vocabulary table to obtain leaf node representation through an embedding matrix.

[0022] The non-leaf nodes in the root path are characterized, and the root path node characterization is input into an LSTM model to obtain a root path context vector;

[0023] The leaf node characterization and the root path context vector are added to obtain an input characterization.

[0024] Optionally, the grouping attention calculation layer applies a grouping attention mechanism to the input characterization, calculates attention scores between the leaf nodes, and obtains an attention output containing context information, including:

[0025] The grouping attention mechanism linearly maps the input characterization to query, key and value representations, groups the attention heads in the query, key or value representations, and calculates attention scores within the group, wherein the attention scores contain relative position information encoded by a rotation position embedding method.

[0026] A Mask operation is applied to the attention scores, and the attention scores after the Mask operation are used for weighting value representations to obtain an attention output containing context information.

[0027] Optionally, the grouping attention mechanism is grouped on the basis of multi-head self-attention, including grouping one of the query, key and value representations, and making the representations in the same group share a single attention head in the remaining representations.

[0028] Optionally, the feedforward neural network adopts a SwiGLU activation function obtained by combining a Swish activation function and a gated linear unit.

[0029] The present application has the following beneficial effects:

[0030] The code completion model GACC based on the grouping attention mechanism effectively reduces the computational complexity and model parameters of the multi-head self-attention mechanism in the prior art by introducing the grouping attention mechanism, significantly improves the processing efficiency of code completion, and reduces the time required for prediction. At the same time, on the basis of improving efficiency and reducing resource consumption, the present application can still maintain a level comparable to advanced models in key performance indicators, providing a more efficient solution for code completion. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0032] Figure 1 A schematic diagram of a rotating position embedding in a Transformer model of an embodiment of the present application;

[0033] Figure 2 A schematic diagram of the overall architecture of a GACC model of an embodiment of the present application;

[0034] Figure 3 A schematic diagram of an input representation construction process of an embodiment of the present application;

[0035] Figure 4 A schematic diagram of a group attention mechanism of an embodiment of the present application under different group numbers;

[0036] Figure 5 A schematic diagram of a code snippet and AST tree leaf node and root path of an embodiment of the present application;

[0037] Figure 6 A TPS index comparison graph under different group numbers of an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0040] The present embodiment provides a code completion method based on group attention, comprising:

[0041] Obtaining code data to be completed;

[0042] The code data to be completed is input into a preset code completion model, and the completed code data is output. The code completion model is trained based on a training set, which includes several open-source code data. The code completion model is used to construct the input code data into an input representation, and then process it through a group attention mechanism and a feedforward neural network to complete the code completion prediction output.

[0043] Specifically, the overall framework of the code completion model GACC proposed in this embodiment is as follows: Figure 2 As shown, it includes four main parts: 1) Input representation construction layer; 2) Group attention calculation layer; 3) Feedforward network processing layer; 4) Prediction output layer. Figure 3 As shown, the input representation construction layer is responsible for fusing the input leaf node sequence with the corresponding root path information, performing lexicalization and embedding processing on the leaf nodes, and extracting context vectors from the root path sequence using an LSTM model. These context vectors are then combined with the leaf node embeddings to generate an input representation containing rich structural and contextual information. The constructed input representation is then fed into an encoding structure consisting of six Transformer layers, with each Transformer layer connected sequentially, meaning the output of the previous layer serves as the input to the next. Each Transformer block contains a group attention mechanism and a feedforward neural network. The detailed principle of the group attention mechanism in this embodiment is as follows... Figure 4 As shown, this mechanism aims to efficiently capture the relationships between leaf nodes in the input sequence. It achieves this by linearly mapping the input representation to obtain query, key, and value representations, and then grouping the attention heads for each query, key, or value representation. Within each group, all attention heads can independently and in parallel compute attention scores, which include information obtained through methods such as... Figure 1 The Rotationally Position Embedded (RoPE) method is used to encode relative position information. Subsequently, a Mask operation is applied to the attention score to ensure that each leaf node only focuses on tokens preceding its current position in the input sequence. The masked attention score is used for weighted representation, resulting in an attention output that incorporates contextual information. Through group sharing, this group attention mechanism reduces the number of parameters and computational complexity, significantly improving the efficiency of processing long input sequences. The output of the group attention is then fed into a feedforward neural network, which employs the SwiGLU activation function, which has stronger nonlinear modeling capabilities, to further learn and represent complex features. After sequential processing through six Transformer blocks, the output of the model's final layer is fed into the prediction output layer, a linear layer that maps the model's internal representation to a probability distribution on a predefined vocabulary, ultimately completing the code completion prediction output.

[0044] The following is a detailed explanation:

[0045] First, the original code data is preprocessed to convert it into a standard format that the model can recognize and process, and the structural information that is helpful for the code completion task is extracted as model training data. The preprocessing process includes the following steps:

[0046] (1) The open source code dataset, such as Python and JavaScript, is parsed to generate the corresponding abstract syntax tree (AST).

[0047] (2) Since the AST in the dataset is usually large in size, the large AST is divided into multiple sub-trees using techniques such as sliding window, and the leaf nodes with "value" and the non-leaf nodes without "value" are identified from the sub-trees, while the root path corresponding to each leaf node is determined.

[0048] (3) Frequency statistics are performed on all nodes of the AST in the dataset, and the most frequent nodes are selected to construct a vocabulary table containing special tokens for unknown nodes "unk_token" and padding "pad_token".

[0049] (4) According to the constructed vocabulary table, the leaf node sequence and the non-leaf node sequence in its corresponding root path are tokenized to convert them into a numerical index sequence that the model can process, such as Figure 5 .

[0050] The above preprocessing procedure is mainly for the model training phase. In the actual application phase after the model is trained, the processing procedure will be simplified accordingly. Specifically, when the model receives the code fragment to be completed, it first parses it into an abstract syntax tree. Then, according to the input format requirements determined in the training phase, the code part closest to the completion position and not exceeding the maximum input limit of the model is intercepted. In this process, the sub-tree division operation does not need to be re-executed. The tokenization process also directly calls the vocabulary table constructed and fixed in the training phase for mapping, without the need to re-construct the vocabulary table.

[0051] The overall processing procedure of the GACC model proposed in this embodiment consists of four main parts:

[0052] I. Input representation construction layer;

[0053] The input representation construction process is shown in Figure 3 , which aims to convert discrete code information into continuous vector representation that the model can understand, and incorporate structural context information. This process includes the following steps:

[0054] (1) For the input code fragment, the leaf node sequence is obtained by parsing. For each leaf node , according to its index in the vocabulary, through a trainable embedding matrix convert it into a low-dimensional, dense leaf node representation vector , the formula is as follows, where represents the vector corresponding to the index found in the embedding matrix , is the dimension of the embedding vector.

[0055] (1) ;

[0056] (2) Extract the root path information related to each leaf node. For leaf node , trace its ancestor nodes in the AST until the root node to form a root path , where is the th non-leaf node on the root path, is the length of the root path. The non-leaf nodes in this root path are also embedded according to the vocabulary, and the root path node embedding sequence is input into the LSTM. The output of the last time of the LSTM is taken as the root path context vector , which will contain the overall semantic information of the root path. The formula is as follows, where represents the root path node embedding sequence:

[0057] (2) ;

[0058] (3) Element-wise addition of the resulting leaf node representation vector and the corresponding root path context vector , to obtain an input representation vector that integrates the leaf node's own semantics and its structural context information, which is used as the input of the GACC model of the present embodiment:

[0059] (3) ;

[0060] The input representation vector sequence constructed will be used as the input of the GACC model. Each vector integrates the leaf node's own information and its structural context information. This input sequence is used as the initial input of the first Transformer block. The input of the subsequent Transformer blocks is the output of the previous block, where .

[0061] II. Group attention calculation layer;

[0062] This embodiment uses a group attention mechanism instead of the multi-head self-attention in the traditional Transformer to efficiently extract the interaction information between leaf nodes in the input sequence. Within each Transformer block, the input representation... First, a linear transformation without bias is applied to map the query, key, and value spaces, where... , , It is a trainable weight matrix:

[0063] , , (4);

[0064] The group attention mechanism used in this embodiment is as follows: Figure 4 As shown, this includes forms such as GQA, GKA, and GVA. This mechanism significantly reduces computation and model parameters while maintaining information exchange by grouping attention heads from either the query, key, or value representation, allowing attention heads within the same group to share a single attention head from the remaining representation. For example, when the number of groups is... In this case, if the keys are grouped (GKA), the number of attention heads for the keys will be grouped, while the number of attention heads for the query and values ​​will be adjusted. At the same time, such as Figure 1 As shown, to incorporate positional information when calculating attention scores, this embodiment employs Rotated Position Embedding (RoPE). For positions where... and The input of the token represents and Query vector after applying RoPE and key vector As follows, among which Represents the rotation embedding matrix. :

[0065] (5);

[0066] (6);

[0067] (7);

[0068] (8);

[0069] Taking key-based grouping (GKA) as an example, for the input representation and The attention score calculation includes the dot product of the query and the key with relative positional information, and considers group sharing:

[0070] (9);

[0071] where is the total number of attention heads, is the number of groups, is the number of attention heads for each group of keys, is the weight matrix corresponding to the query, is the weight matrix corresponding to the th group of keys. The value is represented as . The attention scores are masked to avoid focusing on future information, and then the attention scores are Softmax normalized to obtain the attention weights Finally, the attention weights are weighted summed with the value represented as to obtain the grouped attention output , the calculation formula is as follows:

[0072] (10);

[0073] (11);

[0074] (12);

[0075] Three, feedforward network processing layer;

[0076] After obtaining the output of the grouped attention mechanism, the model will send it to a feedforward neural network for further nonlinear processing and feature conversion. The feedforward network is usually composed of two linear transformation layers with an activation function in between. In this embodiment, the SwiGLU activation function is used in the feedforward network, which has been proven in the field of deep learning to effectively improve the performance and quality of the model. The SwiGLU function is a combination of the Swish activation function and the Gated Linear Unit (GLU), which has stronger nonlinear expression ability than traditional activation functions, helping the model to learn and mine deeper feature representations and complex nonlinear relationships from the attention output. The formula is as follows, where is the Sigmoid activation function, represents the element product.

[0077] (13);

[0078] (14);

[0079] Four, prediction output layer;

[0080] After The final output of the model after processing by a stack of Transformer blocks containing group attention and feed-forward networks is mapped to a vector of vocabulary size by a linear layer. Each element in the vector represents the probability of the corresponding word in the vocabulary as the next code token prediction. The model outputs the prediction result according to this probability distribution, completing the code completion task.

[0081] The code completion model proposed in this embodiment is experimentally verified, and the following describes the data set and experimental settings, evaluation indicators, comparative methods, and experimental results in detail.

[0082] Data set and experimental settings:

[0083] A real-world code dataset is used for experiments to obtain reliable experimental results. Experiments are conducted on two public datasets, Python and JavaScript. Each dataset contains approximately 150,000 abstract syntax trees (ASTs) of code snippets. From them, 100,000 samples are selected as the training set, and 50,000 samples are selected as the test set. In order to process large-scale ASTs, a sliding window approach is used to divide the large ASTs into multiple sub-trees, with the maximum length of the sub-trees limited to . At the same time, in order to utilize the syntax structure information of the AST, LSTM is used to extract the root path information with a length of closest to the leaf node. The size of the constructed vocabulary , including the unknown word "unk_token" and the padding word "pad_token", and the training set token coverage is greater than 90%. In terms of experimental settings, the parameter settings of the PathTrans model and its variants are consistent with the model proposed in this embodiment to ensure the fairness and effectiveness of the comparative experiments.

[0084] Evaluation indicators:

[0085] Two types of evaluation indicators are used to comprehensively evaluate the performance and efficiency of the model. One is the performance indicator, including Top-k accuracy and Mean Reciprocal Rank (MRR), and the other is the efficiency indicator, i.e., the average time required for code completion per batch of samples (TPS).

[0086] (1) Top-k Accuracy: Top-k accuracy is used to measure the ability of the model to include correct candidates in the top k positions in the generated code completion candidate list. It is defined as the ratio of the number of correctly predicted items in the top k positions to the total number of instances. The calculation formula is as follows, where is the number of samples, represents the index list obtained by sorting the probability distribution output by the model in ascending order, is a function to determine whether the true label exists in the list. This embodiment considers different values to evaluate the performance of the model at different recommendation depths.

[0087] (15);

[0088] (16);

[0089] (2) Mean Reciprocal Rank (MRR): MRR evaluates the ranking quality of the model by considering the position of the first correct prediction item in the ranking list. For each instance, the reciprocal of the ranking of the first correct prediction item is calculated, and then the average of the reciprocal rankings of all instances is calculated. The MRR formula is as follows, where denotes the size of the completion list, denotes the ranking of the correct token in the completion list. In order to be closer to the actual use scene of the developer, this embodiment sets to .

[0090] (17);

[0091] (3) Time per sample (TPS): TPS is an efficiency indicator of the model, which measures the average time required by the model to process each batch of samples for code completion. The calculation formula of TPS is as follows, where is the total time required for code completion of all samples, is the total number of samples, is the batch size:

[0092] (18);

[0093] Comparison method:

[0094] In order to verify the effectiveness and efficiency of the GACC model, it is compared with several representative baseline models. All baseline models use the same input representation as the GACC model. The comparison models include:

[0095] (1) Vanilla LSTM: This is a standard LSTM model. Although LSTM has achieved some success in processing sequence data, its sequential processing characteristics limit its parallel computing capability.

[0096] (2) Attention LSTM: This model combines the standard LSTM model and the multi-head self-attention mechanism. It takes the output of LSTM at each time step as the input of multi-head self-attention, and introduces attention mechanism to alleviate the problem of LSTM in processing long sequences.

[0097] (3) PathTrans: A code completion model based on multi-head self-attention. Including the following variants:

[0098] a) PathTrans MHA,R : On the basis of PathTrans based on multi-head self-attention, the PathTrans model of is also used.

[0099] b) PathTrans MQA,R : On the basis of PathTrans based on multi-query attention, the PathTrans model of is also used.

[0100] Experimental results and analysis:

[0101] Table 1 shows the experimental results of the three grouping attention models proposed in this embodiment on Python and JavaScript datasets when the number of groups is 1. Table 2 shows the comparison of GACC-k model with other models.

[0102] Table 1

[0103]

[0104] Table 2

[0105]

[0106] As shown in Table 1 and Table 2, when the number of groups is 1, GACC-q and GACC-k perform similarly and reach a high level in performance indicators such as Top-k and MRR, while GACC-v performs relatively poorly. This shows that GACC-q and GACC-k can more effectively capture complex associations and dependencies in code completion tasks through information sharing between queries and keys, while GACC-v may be relatively limited in performance due to strong information independence. Compared with traditional LSTM-based Vanilla LSTM and AttentionLSTM models, GACC-k achieves better performance on both Python and JavaScript datasets, confirming the superior ability of the Transformer architecture and GACC-k model in capturing long-term dependencies. Compared with the PathTrans model based on Transformer, GACC-k also performs better in Top-k and MRR indicators, mainly due to the effective improvement of the model performance by the rotation position embedding (RoPE) introduced in the attention mechanism of GACC-k. It is worth noting that compared with the best-performing PathTrans MHA,R and PathTrans MQA,R models, the difference in Top-k and MRR indicators of GACC-k is small, which effectively verifies that the code completion model proposed in this embodiment can be comparable in performance to the multi-head attention-based code completion model.

[0107] Recommendation efficiency comparison:

[0108] In addition to the performance of code completion, the proposed method will also be evaluated from the perspective of efficiency. Table 3 shows the comparison of various evaluation indicators of the code completion models of the group attention model GACC-k and PathTrans MQA,R and PathTrans MHA,R on the Python and JavaScript test sets. The experimental results on efficiency are shown in Figure 6 . It can be seen that the TPS of GACC-k on the Python and JavaScript datasets is lower than that of PathTrans MHA,R and PathTrans MQA,R . For example, on the Python dataset, the TPS of GACC-k is about 12.16% lower than that of PathTrans MHA,R , and about 10.57% lower than that of PathTrans MQA,R . On the JavaScript dataset, the TPS of GACC-k is about 9.95% lower than that of PathTrans MHA,R , and about 10.57% lower than that of PathTrans MQA,Rabout 9.37%. In addition, Table 3 also shows that the model size and the number of parameters of GACC-k are reduced compared to PathTrans MQA,R and PathTrans MHA,R are also reduced. These results show that the model based on group attention is more efficient than the models based on multi-head attention and multi-query attention in terms of code completion efficiency.

[0109] Table 3

[0110]

[0111] Hyperparameter comparison analysis:

[0112] Single variable experiments were conducted on the following hyperparameters to explore their respective effects on the performance of the GACC model: embedding dimension, self-attention layer dimension, deep neural network layer number, and group number.

[0113] (1) Effect of embedding dimension: Adjust the embedding dimension size within a certain range. The embedding layer aims to convert high-dimensional input vectors into low-dimensional vectors to be processed, and its size affects the amount of information obtained by the model from the input vector. Generally speaking, a larger embedding size can obtain more abundant information, but an excessively large embedding size increases the amount of data to be processed by the model and cannot achieve data dimension reduction, which may reduce the performance of the model. From the experimental results, it can be seen that as the embedding size increases, the performance of the model improves, and when the embedding size reaches 64, the GACC model achieves the best output in the experiments on the data set, and further increasing the embedding size cannot obtain significant performance improvement.

[0114] (2) Effect of self-attention layer dimension: To explore the effect of the self-attention layer dimension size, the present embodiment adjusts this parameter within a certain range. When applying self-attention to the embedding vector, the self-attention layer dimension size affects the expression ability of the model. Generally speaking, a too small self-attention layer dimension will lead to insufficient self-attention information mining, and even cause the model to perform worse than the model without self-attention. A too large self-attention layer dimension may lead to excessive mining of self-attention information, generating invalid information, and easily leading to overfitting. From the experimental results, it can be seen that in the experiments, when the self-attention layer dimension is 64, GACC achieves relatively the best performance.

[0115] (3) The influence of the number of layers of the deep neural network on the performance of the model is explored. The depth of the model is directly related to its ability to capture and mine high-order interaction features. Specifically, too few layers may result in insufficient mining of high-order interaction features; on the contrary, too many layers will significantly increase the complexity and training time of the model, and may lead to over-mining of high-order interaction features, and thus cause the model to overfit. From the experimental results, it can be seen that when the number of layers of the neural network is 3, the model achieves the best performance.

[0116] (4) The influence of the number of groups: the number of groups of the grouped attention is studied on the performance and efficiency of the model. In the case of , experiments were conducted on the GACC-q, GACC-k and GACC-v models, and the results are shown in Tables 4 and 5. As shown in Table 4, on the Python and JavaScript datasets, the differences in performance indicators such as Top-k and MRR between GACC-q, GACC-k and GACC-v are not significant as the number of groups g increases. However, as shown in Figure 6 , the efficiency indicator TPS of the model significantly increases as g increases. This shows that simply increasing the number of groups does not help improve the performance of the model, but significantly reduces the efficiency of the model. In addition, as shown in Table 5, when the grouped attention is applied to the AttentionLSTM model based on LSTM, the performance improvement is not as significant as in the Transformer architecture.

[0117] Table 4

[0118]

[0119] Table 5

[0120]

[0121] In summary, the code completion model GACC based on grouped attention proposed in the embodiment can improve the efficiency of code completion. Compared with the existing code completion method based on multi-head self-attention, the GACC model proposed in the embodiment has the following advantages: (1) by introducing the grouped attention mechanism, the problem of low efficiency caused by the quadratic relationship between the calculation complexity of multi-head self-attention and the sequence length in the existing code completion model based on Transformer is effectively solved; (2) the grouped attention mechanism processes the attention heads in groups, reduces the number of parameters and the calculation complexity of the model through the sharing mechanism, thereby significantly improving the efficiency of the code completion model while maintaining the performance of the model; (3) the model combines the advanced position encoding method of rotating position embedding, which can effectively integrate the position information of the sequence in the attention calculation process, and enhance the understanding ability of the model to the structure and order of the code sequence. The experimental results on the mainstream programming language dataset show that the GACC model can effectively reduce the time required for the code completion task while maintaining the Top-k accuracy and MRR performance indicators comparable to the model based on multi-head self-attention.

[0122] The above-described embodiments are merely descriptions of the preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for code completion based on packet attention, characterized in that, The method comprises the following steps: acquiring code data to be completed; inputting the code data to be completed into a preset code completion model to output completed code data, wherein the code completion model is obtained based on a training set, the training set comprises a plurality of open source code data, and the code completion model is used to construct input code data into input representation, process the input representation through a grouping attention mechanism and a feedforward neural network, and complete code completion prediction output.

2. The packet attention-based code completion method of claim 1, wherein, Before training the code completion model based on the open source code data in the training set, preprocessing is further performed, and the preprocessing comprises the following steps: parsing the open source code data to generate a corresponding abstract syntax tree; splitting the abstract syntax tree into a plurality of sub-trees, identifying leaf nodes and non-leaf nodes from the sub-trees, and determining a root path corresponding to each leaf node; statistically counting all nodes of the abstract syntax tree, and selecting the node with the highest frequency to construct a vocabulary table, wherein the vocabulary table contains special markers for representing unknown nodes and padding; vocabularyizing leaf node sequences and non-leaf node sequences in their corresponding root paths according to the vocabulary table, and converting the digital index sequences. 3.The packet attention-based code completion method of claim 1, wherein, The code completion model comprises: an input representation construction layer for fusing leaf node sequences and corresponding root path information of input code data to construct input representation; a grouping attention calculation layer for applying a grouping attention mechanism to input representation, calculating attention scores between leaf nodes, and obtaining attention output containing context information; a feedforward network processing layer for inputting grouping attention output into a feedforward neural network to learn and represent complex nonlinear relationships; a prediction output layer for mapping the output of the feedforward neural network to the size of the vocabulary table through a linear layer, calculating the probability of each word in the vocabulary table as a target node, and outputting the prediction result of code completion.

4. The packet attention-based code completion method of claim 3, wherein, The input representation construction layer fuses leaf node sequences and corresponding root path information of input code data to construct input representation, which comprises the following steps: indexing leaf nodes in the vocabulary table to obtain leaf node representation through an embedding matrix; representing non-leaf nodes in the root path, inputting root path node representation into an LSTM model to obtain root path context vectors; adding leaf node representation and root path context vectors to obtain input representation.

5. The packet attention-based code completion method of claim 3, wherein, The grouping attention calculation layer applies a grouping attention mechanism to input representation, calculates attention scores between leaf nodes, and obtains attention output containing context information, which comprises the following steps: The grouping attention mechanism linearly maps the input representation to obtain query, key and value representations, groups attention heads in the query, key or value representations, and calculates attention scores within the groups, wherein the attention scores contain relative position information encoded by a rotation position embedding method; applying a Mask operation to the attention scores, using the attention scores after the Mask operation as weight representations to obtain attention output containing context information.

6. The packet attention-based code completion method of claim 3, wherein, The grouped attention mechanism groups multiple heads of self-attention, including grouping one of the query, key and value representations, and making the grouped representation share a single head of the remaining representations among all heads in the same group.

7. The packet attention-based code completion method of claim 3, wherein, The feedforward neural network adopts a SwiGLU activation function, which is obtained by combining a Swish activation function and a gated linear unit.

Images