Code completion model training method, device, electronic device and storage medium
Through the combination of post-sequence traversal and hierarchical sequences, the problems of information loss and redundant paths in AST modeling in the prior art are solved, the accuracy and accuracy of the code completion model are improved, and it is suitable for modifying code scenarios.
Patent Information
- Application Number
- CN202411134182.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-19
AI Technical Summary
In the training of existing code completion models, the pre-order traversal method loses AST structure information, and the path decomposition generates redundant paths, resulting in a decrease in model training accuracy and accuracy.
The post-sequence traversal method is used to obtain the post-sequence sequence data set of AST, construct the vocabulary and hierarchical sequence data set, AST modeling is performed through post-sequence index sequence and hierarchical sequence, and the code completion model is trained.
It improves the prediction accuracy and accuracy of the code completion model, and is suitable for modifying code scenarios and comprehensively utilizing node context and structural information.
Smart Images

Figure CN118656064B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of code completion, and in particular to a code completion model training method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of machine learning and artificial intelligence technologies, code completion models are being used more and more widely. Using code completion models for code prediction can improve software development efficiency and reduce programming errors. At present, in the process of training code completion models, only the pre-order traversal method is used to model the Abstract Syntax Tree (AST), which will lose AST structure information. The path decomposition method is used to model AST, which generates more redundant paths. The structure-based traversal method is used to model AST, which introduces additional symbols, causing the length of the AST sequence to double, affecting the precision and accuracy of model training. Summary of the invention
[0003] The present invention provides a code completion model training method, device, electronic device and storage medium, so as to solve the defects of the prior art that, in the process of code completion model training, only a pre-order traversal method is used for AST modeling, which will lose AST structure information, a path decomposition method is used for AST modeling, and more redundant paths are generated, and a structure-based traversal method is used for AST modeling, which introduces additional symbols and causes the length of the AST sequence to double, thereby affecting the precision and accuracy of model training.
[0004] In a first aspect, the present invention provides a code completion model training method, comprising:
[0005] Get the open source abstract syntax tree AST dataset to be completed;
[0006] Post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set;
[0007] Building a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary to generate a corresponding post-order index sequence data set;
[0008] Determine a hierarchical sequence data set corresponding to the AST data set;
[0009] Based on the post-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
[0010] In some embodiments, the post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set includes:
[0011] Post-order traversal of each AST data in the AST data set to obtain a post-order traversal sequence of each AST data;
[0012] Using a sliding window mechanism, the post-order traversal sequence of each AST data is segmented to obtain a post-order traversal subsequence set corresponding to the AST data set;
[0013] The post-order traversal subsequence set is screened to obtain a post-order sequence data set corresponding to the AST data set.
[0014] In some embodiments, the step of constructing a vocabulary based on the AST dataset, converting the post-order sequence dataset based on the vocabulary, and generating a corresponding post-order index sequence dataset includes:
[0015] Traversing the AST data set, recording the tags of all AST nodes corresponding to the AST data set, assigning a unique index to each tag, and storing each tag and the index corresponding to each tag in a vocabulary table;
[0016] Based on the vocabulary, an index corresponding to each post-order sequence data in the post-order sequence data set is determined to generate a post-order index sequence data set.
[0017] In some embodiments, determining the hierarchical sequence dataset corresponding to the AST dataset includes:
[0018] Using a depth-first algorithm to traverse each AST data in the AST data set to obtain a hierarchical sequence of each AST data;
[0019] Using a sliding window mechanism, the hierarchical sequence of each AST data is segmented to obtain a hierarchical subsequence set corresponding to the AST data set;
[0020] The hierarchical subsequence set is normalized to obtain a hierarchical sequence data set corresponding to the AST data set.
[0021] In some embodiments, the training of the initial code completion model based on the AST modeling data includes:
[0022] Processing the AST modeling data to obtain an input sequence, a hierarchical sequence, and a target sequence, wherein the input sequence includes semantic information and context information corresponding to the AST modeling data, and the hierarchical sequence includes structural information corresponding to the AST modeling data;
[0023] Performing word embedding on the input sequence to obtain a word vector, and performing position embedding on the hierarchical sequence to obtain a position vector;
[0024] The word vector and the position vector are merged to obtain a hidden state vector;
[0025] Inputting the hidden state vector into the initial code completion model to obtain a code prediction result output by the initial code completion model;
[0026] Calculating a loss function value according to the code prediction result and the target sequence;
[0027] Iteratively optimizing the parameters of the initial code completion model based on the loss function value to obtain the code completion model;
[0028] The code completion model is evaluated.
[0029] In some embodiments, after obtaining the open source abstract syntax tree AST data set to be completed, the method further includes:
[0030] Pre-order traversal of the AST data set to obtain a pre-order sequence data set corresponding to the AST data set;
[0031] Building a vocabulary based on the AST data set, converting the pre-order sequence data set based on the vocabulary to generate a corresponding pre-order index sequence data set;
[0032] Determine a hierarchical sequence data set corresponding to the AST data set;
[0033] Based on the pre-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
[0034] In some embodiments, the code completion model is a transformer-based large language model, and the code completion model includes an embedding layer, a classification layer, and multiple attention modules.
[0035] In a second aspect, the present invention further provides a code completion model training device, comprising:
[0036] An acquisition unit is used to acquire an open source abstract syntax tree AST dataset to be completed;
[0037] A traversal unit, used for post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set;
[0038] A conversion unit, configured to construct a vocabulary based on the AST data set, and convert the post-order sequence data set based on the vocabulary to generate a corresponding post-order index sequence data set;
[0039] A determination unit, used to determine a hierarchical sequence data set corresponding to the AST data set;
[0040] A training unit is used to obtain AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, train an initial code completion model based on the AST modeling data, and obtain a code completion model after the training is completed.
[0041] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the code completion model training method as described above is implemented.
[0042] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the code completion model training methods described above.
[0043] The code completion model training method, device, electronic device and storage medium provided by the present invention obtain an open source abstract syntax tree AST data set to be completed, traverse the AST data set in post-order, and obtain a post-order sequence data set corresponding to the AST data set, thereby capturing the context information of the node, converting the post-order sequence data set, generating a corresponding post-order index sequence data set, determining a hierarchical sequence data set corresponding to the AST data set, and thereby obtaining the structural information of the node, thereby obtaining AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, and performing code completion model training based on the AST modeling data. The present invention can improve the precision and accuracy of code completion model prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 This is one of the flowcharts of the code completion model training method provided by an embodiment of the present invention;
[0046] Figure 2 is a schematic diagram of pre-order traversal and post-order traversal provided by an embodiment of the present invention;
[0047] Figure 3 It is a schematic diagram of a process of post-order traversal of an AST data set to obtain a post-order sequence data set corresponding to the AST data set provided by an embodiment of the present invention;
[0048] Figure 4 It is a schematic diagram of a process of determining a hierarchical sequence data set corresponding to an AST data set provided by an embodiment of the present invention;
[0049] Figure 5 This is the second flow chart of the code completion model training method provided by an embodiment of the present invention;
[0050] Figure 6 This is one of the flowcharts of training an initial code completion model based on AST modeling data provided by an embodiment of the present invention;
[0051] Figure 7 This is the second flow chart of training the initial code completion model based on AST modeling data provided by an embodiment of the present invention;
[0052] Figure 8 is a structural diagram of a code completion model training device provided by an embodiment of the present invention;
[0053] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] Currently, many AST modeling methods regard the context of the code as the code that strictly appears before the cursor, such as the AST modeling methods of pre-order traversal and path decomposition. However, during the code modification process, the context will change. In this scenario, the context of the code is the code around the cursor. In a real development environment, developers often modify existing code, which requires the model to process more contextual information. In addition, the serialized AST essentially loses some structural information. The following challenges are faced when implementing the AST modeling method:
[0056] 1) The process of pre-order traversal, which is to cut first and then traverse, is not applicable to post-order traversal. Since the AST in the original data set is saved in pre-order sequence, if you cut first and then use post-order traversal, the node cannot be found;
[0057] 2) The pre-order sequence is more likely to exceed the input sequence length limit of the model;
[0058] 3) How to efficiently use overly large AST data.
[0059] These issues can make path sets difficult to use in Transformer models.
[0060] To this end, an embodiment of the present invention provides a code completion model training method, device, electronic device and storage medium, by obtaining an open source abstract syntax tree AST data set to be completed, traversing the AST data set in post-order, obtaining a post-order sequence data set corresponding to the AST data set, converting the post-order sequence data set, generating a corresponding post-order index sequence data set, and determining a hierarchical sequence data set corresponding to the AST data set, so that AST modeling data can be obtained based on the post-order index sequence data set and the hierarchical sequence data set, and code completion model training is performed based on the AST modeling data. The present invention can comprehensively utilize node context information and structural information for AST modeling, is suitable for code modification scenarios, and improves the precision and accuracy of code completion model prediction.
[0061] Figure 1 This is a flow chart of a code completion model training method provided by an embodiment of the present invention. Figure 1 As shown, a code completion model training method is provided, including the following steps: step 110 to step 150. The method flow steps are only used as a possible implementation of the present invention.
[0062] Step 110: Obtain an open source abstract syntax tree AST dataset to be completed.
[0063] Among them, the abstract syntax tree AST is a data structure used to represent source code in programming languages. In the programming context, AST is a tree structure consisting of nodes and edges. Each node represents a grammatical structure in the source code, such as an expression, statement, function definition, etc., and the edge represents the relationship between these grammatical structures.
[0064] Step 120: traverse the AST data set in post-order to obtain a post-order sequence data set corresponding to the AST data set.
[0065] It should be noted that post-order traversal is a depth-first traversal method that visits nodes in the order of "left-right-root". The "left-right-root" access order of post-order traversal means that when each node is visited, its child nodes have been processed. This traversal method perfectly fits the scenario of context prediction when modifying the code, because when visiting each node, the context information of the node has been fully captured, which helps the model better understand the logic and structure of the code.
[0066] Optionally, recursively traverse each child node of the AST data starting from the root node, and then visit the current node after all child nodes of the AST data have been visited, to generate a post-order sequence of the AST data.
[0067] Figure 2 Schematic diagram of pre-order traversal and post-order traversal provided by an embodiment of the present invention. Figure 2 As shown, assuming that node No. 10 is predicted, the pre-order sequence can only obtain the nodes before No. 10, that is, "Call, Name, print, Name", while the post-order sequence can obtain the nodes around No. 10, that is, "print, Name, Call, s".
[0068] Figure 3 A schematic diagram of a process of post-order traversal of an AST data set to obtain a post-order sequence data set corresponding to the AST data set provided by an embodiment of the present invention is as follows: Figure 3 As shown, in some embodiments, post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set includes:
[0069] Post-order traversal of each AST data in the AST data set to obtain the post-order traversal sequence of each AST data;
[0070] Using the sliding window mechanism, the post-order traversal sequence of each AST data is segmented to obtain the post-order traversal subsequence set corresponding to the AST data set;
[0071] The post-order traversal subsequence set is screened to obtain the post-order sequence data set corresponding to the AST data set.
[0072] Optionally, a sliding window is used to cut the post-order traversal sequence of the AST data into shorter segments.
[0073] It should be noted that the size of the sliding window and the distance of each sliding can be set dynamically when the code is executed to reuse the nodes in the AST to varying degrees and increase the model training data.
[0074] For example, the size of the sliding window is 1000 and the distance of each slide is 500.
[0075] Optionally, the post-order traversal subsequence set is screened based on a preset length threshold to obtain a post-order sequence data set corresponding to the AST data set, which can avoid unnecessary calculations or processing, thereby improving data processing efficiency.
[0076] Step 130: construct a vocabulary based on the AST data set, convert the post-order sequence data set based on the vocabulary, and generate a corresponding post-order index sequence data set.
[0077] The vocabulary contains all unique symbols or tokens that appear in the AST dataset. These symbols can be types of syntax nodes, operators, keywords, etc.
[0078] For example, an AST data may contain node types such as "Expression", "Binary Operation", "Function Call", etc., and operators such as "+", "-", " / ", etc.
[0079] In some embodiments, step 130 constructs a vocabulary based on the AST data set, converts the post-order sequence data set based on the vocabulary, and generates a corresponding post-order index sequence data set, including:
[0080] Traverse the AST data set, record the tags of all AST nodes corresponding to the AST data set, assign a unique index to each tag, and store each tag and the index corresponding to each tag in the vocabulary;
[0081] Based on the vocabulary table, the index corresponding to each post-order sequence data in the post-order sequence data set is determined to generate a post-order index sequence data set.
[0082] Optionally, all occurrences of the token in the vocabulary are uniqued and each unique token is assigned an index, typically starting from 0 and increasing in number.
[0083] It can be understood that by converting complex post-order sequence data into simple index sequence data, the representation and processing of data can be greatly simplified. Using post-order index sequence data instead of original post-order sequence data can reduce the complexity of data storage and processing and improve processing efficiency. Post-order index sequence data sets are more suitable for machine learning, especially models that require serialized input, such as recurrent neural networks and Transformer models.
[0084] Step 140: Determine the hierarchical sequence data set corresponding to the AST data set.
[0085] Optionally, each AST data in the abstract syntax tree AST data set is read, and different algorithms are used in different scenarios to obtain a hierarchical sequence, in which the hierarchical sequence elements correspond one-to-one to the post-order traversal sequence elements.
[0086] Figure 4 A schematic diagram of a process for determining a hierarchical sequence dataset corresponding to an AST dataset provided in an embodiment of the present invention. In some embodiments, determining a hierarchical sequence dataset corresponding to an AST dataset includes:
[0087] Use the depth-first algorithm to traverse each AST data in the AST data set to obtain the hierarchical sequence of each AST data;
[0088] The sliding window mechanism is used to segment the hierarchical sequence of each AST data to obtain the hierarchical subsequence set corresponding to the AST data set;
[0089] The hierarchical subsequence set is normalized to obtain the hierarchical sequence dataset corresponding to the AST dataset.
[0090] It should be noted that a hierarchical sequence of AST data may be too long, exceeding the maximum sequence length that can be input into the model. Therefore, the hierarchical sequence needs to be segmented using the same sliding window mechanism as the post-order traversal method. The size of the sliding window and the distance of each sliding are the same as those of the post-order traversal.
[0091] Optionally, the set of hierarchical subsequences is normalized to ensure that the minimum level of each hierarchical subsequence is 1.
[0092] Step 150: Based on the post-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
[0093] In some embodiments, the code completion model is a large language model based on transformer, and the code completion model includes an embedding layer, a classification layer, and multiple attention modules.
[0094] It is understandable that compared with the traditional Recurrent Neural Network (RNN) and Long Short-Term Memory Network (LSTM) models, the transformer-based large language model has the following advantages:
[0095] 1) Parallel computing: The Transformer model can process all positions in the sequence in parallel during training, while the LSTM needs to process each time step sequentially;
[0096] 2) Long-distance dependency: The Transformer model performs better in capturing long-distance dependencies in sequences. The self-attention mechanism allows the Transformer model to directly focus on any position in the sequence, while the memory mechanism of LSTM may decay when dealing with long-distance dependencies.
[0097] 3) Model expressiveness: The Transformer model uses a multi-head self-attention mechanism and a deeper network structure, allowing the model to capture multi-level feature representations;
[0098] 4) Training speed: Since all positions in the sequence can be processed in parallel, the Transformer model is more computationally efficient, while the sequential processing characteristics of LSTM make it slower to train on long sequences.
[0099] In some embodiments, after obtaining the open source abstract syntax tree AST data set to be completed, the process further includes:
[0100] Pre-order traverse the AST data set to obtain the pre-order sequence data set corresponding to the AST data set;
[0101] Build a vocabulary based on the AST data set, transform the pre-order sequence data set based on the vocabulary, and generate the corresponding pre-order index sequence data set;
[0102] Determine the hierarchical sequence dataset corresponding to the AST dataset;
[0103] Based on the pre-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
[0104] Optionally, in a pre-order traversal, the order of visiting nodes is: first visit the root node, then recursively traverse the left subtree, and finally recursively traverse the right subtree.
[0105] Figure 5 The second flowchart of the code completion model training method provided by the embodiment of the present invention is as follows: Figure 5 As shown, in the code writing scenario, pre-order traversal is used to obtain the pre-order traversal sequence data of AST, and in the code modification scenario, post-order traversal is used to obtain the post-order traversal sequence data of AST. Based on the vocabulary, the pre-order traversal sequence data or the post-order traversal sequence data is sequence converted to obtain a pre-order index sequence dataset or a post-order index sequence dataset, and a hierarchical sequence dataset is obtained. The pre-order index sequence dataset or the post-order index sequence dataset and the hierarchical sequence dataset are loaded with data to obtain the model input data, and the code completion model is trained.
[0106] It should be noted that in the code writing scenario, AST can be modeled by using a pre-order traversal sequence plus a hierarchical sequence corresponding to the pre-order nodes. When writing code, the context of the code is strictly limited to the code before the cursor, and the pre-order traversal sequence can naturally reflect the code writing logic and obtain the above information when writing code. The hierarchical traversal sequence is used to supplement the structural information lost by AST serialization.
[0107] It should be noted that in the code modification scenario, AST can be modeled using a post-order traversal sequence and a hierarchical sequence corresponding to post-order nodes. When modifying the code, the context of the code can be the code before and after the cursor, and when using post-order traversal to access each node, the information of the node's context and context can be captured at the same time. The hierarchical traversal sequence is used to supplement the structural information lost by AST serialization.
[0108] Optionally, in different scenarios, the hierarchical sequence visits the nodes in the tree in the same traversal order as the pre-order or post-order traversal, and records the level at which the node is located.
[0109] It should be noted that the hierarchical sequence of AST nodes is used to represent the structural information of AST. The hierarchical sequence is the level corresponding to each node when different AST serialization methods are used, reflecting the hierarchical structure of AST. It should be noted that the hierarchical sequence used in the embodiment of the present invention is not the result of the classic tree level-order traversal, but the result of a depth-first traversal similar to a pre-order or post-order traversal, and the hierarchical sequence records the depth of the node in the tree.
[0110] Optionally, the elements in the pre-order sequence and its level sequence correspond one-to-one, and the order remains consistent; the elements in the post-order sequence and its level sequence correspond one-to-one, and the order remains consistent.
[0111] Optionally, the input is a pre-order index sequence dataset or a post-order index sequence dataset and a hierarchical sequence dataset. The data loader divides the dataset into multiple small batches and inputs one batch of data each time for training, thereby improving the model training efficiency and reducing memory usage. At the same time, the pre-order index sequence dataset or the post-order index sequence dataset is further processed to obtain the input sequence and the target sequence.
[0112] In an embodiment of the present invention, by obtaining an open source abstract syntax tree AST dataset to be completed, traversing the AST dataset in post-order, and obtaining a post-order sequence dataset corresponding to the AST dataset, the context information of the node can be captured, the post-order sequence dataset is converted, and the corresponding post-order index sequence dataset is generated. The hierarchical sequence dataset corresponding to the AST dataset is determined, and the structural information of the node can be obtained, so that the AST modeling data can be obtained based on the post-order index sequence dataset and the hierarchical sequence dataset, and the code completion model is trained based on the AST modeling data. The present invention can improve the precision and accuracy of the code completion model prediction.
[0113] Figure 6 One of the flow diagrams of training an initial code completion model based on AST modeling data provided in an embodiment of the present invention. In some embodiments, training an initial code completion model based on AST modeling data includes:
[0114] Processing the AST modeling data to obtain an input sequence, a hierarchical sequence, and a target sequence, wherein the input sequence includes semantic information and context information corresponding to the AST modeling data, and the hierarchical sequence includes structural information corresponding to the AST modeling data;
[0115] Perform word embedding on the input sequence to obtain word vectors, and perform position embedding on the hierarchical sequence to obtain position vectors;
[0116] Merge the word vector and the position vector to get the hidden state vector;
[0117] Inputting the hidden state vector into the initial code completion model to obtain the code prediction result output by the initial code completion model;
[0118] Calculate the loss function value based on the code prediction results and the target sequence;
[0119] Based on the loss function value, the parameters of the initial code completion model are iteratively optimized to obtain the code completion model;
[0120] Evaluate code completion models.
[0121] Among them, the word vector captures the semantic and contextual information in the AST data, and the position vector captures the structural information in the AST data.
[0122] Optionally, back propagation and gradient descent algorithms are used to update model parameters.
[0123] Optionally, the mean reciprocal rank (MRR) is calculated according to the code prediction results and the target sequence, and the code completion model is evaluated based on the MRR.
[0124] Among them, MRR is calculated by taking the average of the reciprocal ranking of each prediction result (that is, the reciprocal ranking of the first correct answer). Therefore, a higher MRR means a higher prediction accuracy.
[0125] Table 1 shows the MRR of the pre-order sequence AST modeling method and the pre-order sequence plus hierarchical sequence AST modeling method when predicting different types of leaf nodes. As shown in Table 1, in the scenario of writing code, the pre-order sequence plus hierarchical sequence AST modeling method is used to construct AST data and perform model training, which can improve the accuracy of the model's prediction of various types of leaf nodes.
[0126] Table 1 MRR of the preorder sequence AST modeling method and the preorder sequence plus hierarchical sequence AST modeling method in predicting different types of leaf nodes
[0127]
[0128] Table 2 shows the MRR of the pre-order sequence AST modeling method and the post-order sequence plus hierarchical sequence AST modeling method when predicting the next token of different types. As shown in Table 2, the post-order sequence plus hierarchical sequence AST modeling method is used to construct AST data and perform model training, which can improve the accuracy of the model's prediction of various types of leaf nodes.
[0129] Table 2 MRR of the pre-order sequence AST modeling method and the post-order sequence plus hierarchical sequence AST modeling method when predicting the next token of different types
[0130]
[0131] It should be noted that the MRR in Table 1 represents the prediction accuracy of the leaf (value) node, while the MRR in Table 2 represents the average prediction accuracy of the leaf (value) node and its parent node (type). Therefore, the MRRs of the AST modeling method using both preorder sequences are different.
[0132] Table 3 shows the MRR of the pre-order sequence AST modeling method and the post-order sequence plus hierarchical sequence AST modeling method when predicting different types of nodes. As shown in Table 3, in the code modification scenario, the use of the post-order sequence and the corresponding hierarchical sequence AST modeling method can improve the prediction accuracy of the model.
[0133] Table 3 MRR of the pre-order sequence AST modeling method and the post-order sequence plus hierarchical sequence AST modeling method in predicting different types of nodes
[0134]
[0135] Figure 7 The second flow chart of training the initial code completion model based on AST modeling data provided by the embodiment of the present invention is as follows: Figure 7 As shown, based on AST data, the context-aware serialization (CAS) method is used for AST modeling. Based on the AST modeling data, the node context information and structure information are extracted, the node representation in the CAS sequence is converted into an embedding vector, and the node context information and hierarchy indicator are combined to construct input and position embeddings. For each node, its context information and hierarchy indicator are fused with the embedding vector to obtain the final input vector. By inputting the input vector into the model, the code distribution of all possible tags is obtained.
[0136] Among them, the CAS method uses post-order sequences and corresponding hierarchical sequences to model AST.
[0137] It should be noted that the context information of the node is another key element in the CAS method. During the serialization process, not only the information of each node itself is recorded, but also its context information, including parent nodes, child nodes, etc. This context information can help the model better understand the position and role of the node in the AST.
[0138] For example, in a post-order traversal, when each node is visited, while retaining the information of the node before it, the information of its child nodes has been recorded, so the information of these child nodes can be included when generating the post-order sequence.
[0139] For example, the known code snippet includes not only the first t-1 tokens: C1={c1,c2,…,ct-1}, but also the last n tokens: C2={ct+1, ct+2,…,ct+n}. At this time, the conditional probability formula for the sequence prediction problem is:
[0140] ;
[0141] ci represents the token in AST, It means finding the probability of the next token when some tokens are known.
[0142] It should be noted that structural information is used to identify and represent the structural hierarchy of AST. When generating a hierarchical sequence, the hierarchical position of each node is used to identify its structural hierarchy, which can help the model distinguish nodes of different levels and understand the hierarchical relationship between nodes. Not only the parent-child relationship of different levels, but also the sibling relationship of the same level will be recorded. These are used to enrich the context of the node, making the final generated hierarchical sequence more semantically coherent.
[0143] Optionally, in position embedding, hierarchical identifiers are used to enhance the hierarchical representation of nodes and make up for some structural information lost in AST serialization. The calculation formula for position embedding is as follows:
[0144] lv=ASTposition_embedding(v);
[0145] In the above formula, lv is the position encoding of node v in AST, and ASTposition_embedding is a function that maps hierarchical sequence data to vector space.
[0146] Among them, the Transformer model consists of multiple attention blocks (AttnBlk), which are preceded by an input embedding layer (Emb) and followed by a classification layer (Clsfr). The attention block (AttnBlk) is repeated n times.
[0147] Optionally, the code predicts an expression like this:
[0148] Trans(hv,lv) = Clsfr(AttnBlk(...(AttnBlk(hv + lv))...));
[0149] Where hv=ASTinputembedding(v), hv is the embedding representation of node v in AST, and ASTinputembedding is a function that maps AST post-order traversal data to vector space.
[0150] The following is a description of a code completion model training device provided in an embodiment of the present invention. The code completion model training device described below and the code completion model training method described above can refer to each other.
[0151] Figure 8 A schematic diagram of the structure of a code completion model training device provided by an embodiment of the present invention, such as Figure 8 As shown, the code completion model training device 800 includes:
[0152] An acquisition unit 810 is used to acquire an open source abstract syntax tree AST data set to be completed;
[0153] A traversal unit 820 is used to traverse the AST data set in post-order to obtain a post-order sequence data set corresponding to the AST data set;
[0154] A conversion unit 830 is used to construct a vocabulary based on the AST data set, and convert the post-order sequence data set based on the vocabulary to generate a corresponding post-order index sequence data set;
[0155] A determination unit 840 is used to determine a hierarchical sequence data set corresponding to the AST data set;
[0156] The training unit 850 is used to obtain AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, train the initial code completion model based on the AST modeling data, and obtain the code completion model after the training is completed.
[0157] Optionally, the AST dataset is traversed in post-order to obtain a post-order sequence dataset corresponding to the AST dataset, including:
[0158] Post-order traversal of each AST data in the AST data set to obtain the post-order traversal sequence of each AST data;
[0159] Using the sliding window mechanism, the post-order traversal sequence of each AST data is segmented to obtain the post-order traversal subsequence set corresponding to the AST data set;
[0160] The post-order traversal subsequence set is screened to obtain the post-order sequence data set corresponding to the AST data set.
[0161] Optionally, a vocabulary is constructed based on the AST data set, and the post-order sequence data set is transformed based on the vocabulary to generate a corresponding post-order index sequence data set, including:
[0162] Traverse the AST data set, record the tags of all AST nodes corresponding to the AST data set, assign a unique index to each tag, and store each tag and the index corresponding to each tag in the vocabulary;
[0163] Based on the vocabulary table, the index corresponding to each post-order sequence data in the post-order sequence data set is determined to generate a post-order index sequence data set.
[0164] Optionally, determine the hierarchical sequence dataset corresponding to the AST dataset, including:
[0165] Use the depth-first algorithm to traverse each AST data in the AST data set to obtain the hierarchical sequence of each AST data;
[0166] The sliding window mechanism is used to segment the hierarchical sequence of each AST data to obtain the hierarchical subsequence set corresponding to the AST data set;
[0167] The hierarchical subsequence set is normalized to obtain the hierarchical sequence dataset corresponding to the AST dataset.
[0168] Optionally, an initial code completion model is trained based on AST modeling data, including:
[0169] Processing the AST modeling data to obtain an input sequence, a hierarchical sequence, and a target sequence, wherein the input sequence includes semantic information and context information corresponding to the AST modeling data, and the hierarchical sequence includes structural information corresponding to the AST modeling data;
[0170] Perform word embedding on the input sequence to obtain word vectors, and perform position embedding on the hierarchical sequence to obtain position vectors;
[0171] Merge the word vector and the position vector to get the hidden state vector;
[0172] Inputting the hidden state vector into the initial code completion model to obtain the code prediction result output by the initial code completion model;
[0173] Calculate the loss function value based on the code prediction results and the target sequence;
[0174] Based on the loss function value, the parameters of the initial code completion model are iteratively optimized to obtain the code completion model;
[0175] Evaluate code completion models.
[0176] Optionally, the code completion model training device 800 further includes:
[0177] Pre-order traverse the AST data set to obtain the pre-order sequence data set corresponding to the AST data set;
[0178] Build a vocabulary based on the AST data set, transform the pre-order sequence data set based on the vocabulary, and generate the corresponding pre-order index sequence data set;
[0179] Determine the hierarchical sequence dataset corresponding to the AST dataset;
[0180] Based on the pre-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
[0181] Optionally, the code completion model is a transformer-based large language model, and the code completion model includes an embedding layer, a classification layer, and multiple attention modules.
[0182] It should be noted here that the code completion model training device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned code completion model training method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.
[0183] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the code completion model training method, which includes: obtaining an open source abstract syntax tree AST data set to be completed; post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set; constructing a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary, and generating a corresponding post-order index sequence data set; determining the hierarchical sequence data set corresponding to the AST data set; obtaining AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, training an initial code completion model based on the AST modeling data, and obtaining a code completion model after the training is completed.
[0184] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0185] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the code completion model training method provided by the above methods, which includes: obtaining an open source abstract syntax tree AST data set to be completed; post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set; building a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary, and generating a corresponding post-order index sequence data set; determining the hierarchical sequence data set corresponding to the AST data set; obtaining AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, training an initial code completion model based on the AST modeling data, and obtaining a code completion model after the training is completed.
[0186] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the code completion model training method provided by the above-mentioned methods, the method comprising: obtaining an open source abstract syntax tree AST data set to be completed; traversing the AST data set in post-order to obtain a post-order sequence data set corresponding to the AST data set; constructing a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary, and generating a corresponding post-order index sequence data set; determining a hierarchical sequence data set corresponding to the AST data set; obtaining AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, training an initial code completion model based on the AST modeling data, and obtaining a code completion model after the training is completed.
[0187] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A code completion model training method, characterized in that: include: Get the open source abstract syntax tree AST dataset to be completed; Post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set; Building a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary to generate a corresponding post-order index sequence data set; Determine a hierarchical sequence data set corresponding to the AST data set; Based on the post-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained; The post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set includes: Post-order traversal of each AST data in the AST data set to obtain a post-order traversal sequence of each AST data; Using a sliding window mechanism, the post-order traversal sequence of each AST data is segmented to obtain a post-order traversal subsequence set corresponding to the AST data set; Screening the post-order traversal subsequence set to obtain a post-order sequence data set corresponding to the AST data set; The step of determining the hierarchical sequence data set corresponding to the AST data set includes: Using a depth-first algorithm to traverse each AST data in the AST data set to obtain a hierarchical sequence of each AST data; Using a sliding window mechanism, the hierarchical sequence of each AST data is segmented to obtain a hierarchical subsequence set corresponding to the AST data set; Normalizing the hierarchical subsequence set to obtain a hierarchical sequence data set corresponding to the AST data set; The training of the initial code completion model based on the AST modeling data includes: Processing the AST modeling data to obtain an input sequence, a hierarchical sequence, and a target sequence, wherein the input sequence includes semantic information and context information corresponding to the AST modeling data, and the hierarchical sequence includes structural information corresponding to the AST modeling data; Performing word embedding on the input sequence to obtain a word vector, and performing position embedding on the hierarchical sequence to obtain a position vector; The word vector and the position vector are merged to obtain a hidden state vector; Inputting the hidden state vector into the initial code completion model to obtain a code prediction result output by the initial code completion model; Calculating a loss function value according to the code prediction result and the target sequence; Iteratively optimizing the parameters of the initial code completion model based on the loss function value to obtain the code completion model; The code completion model is evaluated.
2. The code completion model training method according to claim 1, characterized in that: The step of constructing a vocabulary based on the AST data set, converting the post-order sequence data set based on the vocabulary, and generating a corresponding post-order index sequence data set includes: Traversing the AST data set, recording the tags of all AST nodes corresponding to the AST data set, assigning a unique index to each tag, and storing each tag and the index corresponding to each tag in a vocabulary table; Based on the vocabulary, an index corresponding to each post-order sequence data in the post-order sequence data set is determined to generate a post-order index sequence data set.
3. The code completion model training method according to claim 1, characterized in that: After obtaining the open source abstract syntax tree AST data set to be completed, the following steps are also included: Pre-order traversal of the AST data set to obtain a pre-order sequence data set corresponding to the AST data set; Building a vocabulary based on the AST data set, converting the pre-order sequence data set based on the vocabulary to generate a corresponding pre-order index sequence data set; Determine a hierarchical sequence data set corresponding to the AST data set; Based on the pre-order index sequence data set and the hierarchical sequence data set, AST modeling data is obtained, and an initial code completion model is trained based on the AST modeling data. After the training is completed, a code completion model is obtained.
4. The code completion model training method according to any one of claims 1 to 3, characterized in that: The code completion model is a large language model based on transformer, and the code completion model includes an embedding layer, a classification layer and multiple attention modules.
5. A code completion model training device, characterized in that: The device is used to execute the code completion model training method according to any one of claims 1 to 4, comprising: An acquisition unit is used to acquire an open source abstract syntax tree AST dataset to be completed; A traversal unit, used for post-order traversal of the AST data set to obtain a post-order sequence data set corresponding to the AST data set; A conversion unit, configured to construct a vocabulary based on the AST data set, and convert the post-order sequence data set based on the vocabulary to generate a corresponding post-order index sequence data set; A determination unit, used to determine a hierarchical sequence data set corresponding to the AST data set; A training unit is used to obtain AST modeling data based on the post-order index sequence data set and the hierarchical sequence data set, train an initial code completion model based on the AST modeling data, and obtain a code completion model after the training is completed.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the code completion model training method as described in any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the code completion model training method as described in any one of claims 1 to 4 is implemented.
