A code completion method based on structural features and sequence features

By combining structural and sequence features in the code completion model, and utilizing abstract syntax trees and extended attention mechanisms, the problem of insufficient utilization of out-of-vocabulary words and sequence information in existing technologies is solved, thereby improving the accuracy of code completion and model performance.

CN114924741BActive Publication Date: 2024-11-15NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210505898.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-10
Publication Date
2024-11-15
Estimated Expiration
2042-05-10

AI Technical Summary

Technical Problem

Existing technologies fail to effectively utilize the sequence information of code in code completion, ignore the semantic logic and procedural regularity of the code, and do not fully solve the problem of out-of-vocabulary words, resulting in a decrease in model prediction accuracy and performance.

Method used

We employ a method based on structural and sequence features. By converting source code into an abstract syntax tree, we build a vocabulary using an extended attention mechanism and a byte pair encoding algorithm. Combined with a Transformer decoder, we capture the relative structure and positional relationships of the code, thereby improving the learning performance of the model.

Benefits of technology

It enhances the semantic expressive power of the code completion model, reduces the out-of-vocabulary word problem, and improves the accuracy of code completion and the accuracy of candidate results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114924741B_ABST
    Figure CN114924741B_ABST
Patent Text Reader

Abstract

The present invention discloses a code completion method based on structural features and sequence features, comprising a preprocessing part, a code part conversion and a training part; in the preprocessing part, source code is collected from an open source platform and converted into an abstract syntax tree form; a byte pair encoding method is used to establish a corresponding vocabulary; in the code conversion part, an extended attention mechanism is used to extract the structural features and sequence features of the code; finally, a Transfomer decoder is used as a basic network model, the attention mechanism is extended, the probability is calculated through a softmax layer, and the result is output; the present invention introduces a byte pair encoding algorithm to establish a vocabulary for values, so that the problem of unregistered words in the constructed language model is greatly reduced; in addition, by considering the semantic information of the code from two dimensions of the code, the accuracy of code completion is improved, and users are helped to obtain more accurate candidates.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer software engineering, and mainly to a code completion method based on structural features and sequence features. Background Art

[0002] As the scale of software development continues to expand, code completion assisting developers in software development has gained more and more attention. It accelerates the software development process by proposing the next possible markup based on the existing code in the integrated development environment (IDE). It can become a convenient documentation and input method for developers. Code completion has become an important part of many code intelligence tasks and plays an important role in automatic program generation and rapid iteration of software products.

[0003] Neural network-based code completion models usually convert code snippets into source code sequences or abstract syntax trees (ASTs), or graph-based, and then select a suitable neural network to build a language model.

[0004] Most of the existing technologies focus on extracting features from abstract syntax trees and encoding parts of abstract syntax trees to complete the task. The abstract syntax tree not only represents source code and hierarchical information. It also accurately depicts the logic and code structure in the source code. These methods focus on code features on AST, but are not enough in sequence information mining. AST reflects the structure or hierarchy of the code, and the traversal method makes it easy to extract the code structure or the path between nodes. The nodes on the tree can contain arbitrary information, which means that there are more features after converting the code snippet to AST to help improve the model performance. Code sequence representation is the simplest and most effective way to input the model, which can directly represent the semantics of the code snippet, but the structural features of the code are still omitted. Although the syntax of different programming languages ​​may differ, for example, Java uses curly braces to organize a code block, and Python uses indentation to make several statements into a code block, but they all contain sequence, loop and selection structures. These are the basic components of programming languages ​​and the basis for building code blocks. Sequence representation ignores them and processes the code as natural language. Compared with the structural features that focus on depicting the code hierarchy, sequence features contain more potential regularities that reflect the semantic logic and process of the code.

[0005] When building models for source code-based software engineering tasks, one of the key issues is how to model the vocabulary, but only a few methods have considered the unknown word problem. Unlike natural languages, developers can create variable and function names with a combination of words of any length, which inevitably leads to the problem that the vocabulary cannot encode new word strings, called unknown words. Some words are not observed in the training set but appear in the test set, which can seriously degrade the prediction accuracy and performance of the model. Other methods create a vocabulary by retaining frequently occurring words and marking unknown words with a special placeholder, or introduce a cache mechanism or pointer network to alleviate, but do not completely solve this problem. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a code completion method based on structural features and sequence features, which can provide more code features for the neural network model, so that the learning effect is improved to provide more accurate candidates and the completion effect is better.

[0007] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A code completion method based on structural features and sequence features comprises the following steps:

[0009] Step S1, code preprocessing: collecting source code from an open source platform, converting the source code into an abstract syntax tree; using byte pair encoding to establish a corresponding vocabulary;

[0010] Step S2: code conversion; using the extended attention mechanism to extract the structural features and sequence features of the code;

[0011] Step S3, training part; convert the collected source code data into the data format of the abstract syntax tree after the preprocessing described above; send the matrix consisting of the abstract syntax tree sequence and the relative paths of the extracted nodes to the model for calculation; the model uses the Transformer decoder as the basic neural network and extends the attention mechanism, where the extension introduces a new value R when calculating the query Q and the key K, R represents the relative hierarchical traversal path between nodes, so as to capture the relative structural information; introduce a new variable and When calculating the attention score of a node, the score weight with another node is taken into account to extract the relative position relationship of the code sequence; finally, the probability is calculated through the softmax layer and the result is output.

[0012] Furthermore, in step S1, the source code is converted into an abstract syntax tree form; each node on the tree includes two parts: a node type and a node value; when a node on a tree has no node value, it is filled with a placeholder [empty]; in order to ensure that the data in the corpus is available, a series of steps are taken, including removing non-ASCII encoded characters, removing comments, and performing word segmentation on variables named in camel case and underscore; then, a vocabulary is established for the node type and node value respectively, wherein a byte pair encoding method is used to establish a vocabulary for the node value.

[0013] Furthermore, the specific method of constructing a vocabulary for node values ​​using byte pair encoding in step S1 is as follows:

[0014] The byte pair encoding algorithm is an iterative operation: first, each word in the corpus is segmented, and the single characters are added to the initial vocabulary; then the frequency of two consecutive characters is counted, and the character pair with the highest frequency (t1, t2) is merged into a new character (t1t2). This process is called a merge operation; repeat this process until there are no character pairs that can be merged or the number of merge operations reaches a preset value. The final byte pair encoded vocabulary includes all the initial characters, merged character pairs, and mergeable pairs.

[0015] Furthermore, the specific steps of code conversion in step S2 include:

[0016] Step S2.1, converting the abstract syntax tree obtained in step S1 to extract the structural features of the abstract syntax tree; examining each pair of nodes on the tree, finding a unique path by traversing, and all the paths together represent the relative structural information and hierarchical information of the abstract syntax tree;

[0017] Step S2.2: After pre-order traversal, an abstract syntax tree sequence is obtained, and the relative position relationship of each pair of nodes in the sequence is examined to extract sequence features.

[0018] Furthermore, the path representation in step S2.1 is as follows:

[0019] UD path (a,b)=U i D j

[0020] Among them U i represents the number of upward movements from node a to node b, D j Indicates the number of downward movements from node a to node b;

[0021] Introducing matrix R to represent UD in neural network calculation path The matrix formed is:

[0022]

[0023] Where Q, K, and V represent the matrix consisting of the query, key, and value vectors of the input elements. The softmax function is used to calculate the weight that determines the degree of expression of V. In order to avoid the QK product from being too large and causing the gradient to disappear, the result is divided by Alleviate the vanishing gradient problem. are the dimensions along which query, key, and value vectors are created for the input elements.

[0024] Furthermore, the specific method for extracting sequence features in step S2.2 is as follows:

[0025] The code sequence features are expressed as repeated patterns and regularity information. The extracted sequence features are the relative position encoding of the nodes on the sequence, which is specifically expressed as follows:

[0026]

[0027]

[0028] where z i is x i The attention score, W V , W Q , W K is a trainable matrix, a ij is x i For the current position x j The attention are two trainable parameters of the extended attention mechanism; a ij The calculation of is to get the probability distribution through the softmax function.

[0029] Beneficial effects:

[0030] The code completion method based on structural features and sequence features provided by the present invention enhances the semantic expression ability of the code by mining the structural features and sequence features of the code, so that the model can learn more code features. In addition, the present invention also introduces a byte pair encoding algorithm to establish a vocabulary for node values, so that the problem of unregistered words in the constructed language model is greatly reduced. The present invention considers the semantic information of the code from two dimensions of the code, so that the accuracy of code completion is improved, helping users to obtain more accurate candidates. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the code completion method based on structural features and sequence features provided by the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, this embodiment provides a code completion method based on structural features and sequence features, including a preprocessing part, a data conversion part and a training part. Specifically,

[0034] Step S1, code preprocessing stage;

[0035] Compared with natural languages, programming languages ​​are simpler and more fixed because they have rules and fixed syntax. To fully utilize the semantic information hidden in the code, it cannot be processed in only one form. The abstract syntax tree is a common code representation in software engineering. The nodes on the tree can contain any information, which makes it possible to represent more information. This embodiment converts the source code into an abstract syntax tree; each node on the tree includes two parts: a node type and a node value; when a node on a tree has no node value, it is filled with a placeholder [empty].

[0036] In order to solve the problem of unregistered words in the constructed language model, the byte pair encoding method is used in this embodiment to establish a vocabulary for the node value. The byte pair encoding algorithm is an iterative operation: first, each word in the corpus is segmented, and a single character is obtained and added to the initial vocabulary; then the frequency of two consecutive characters is counted, and the character pair with the highest frequency (t1, t2) is merged into a new character (t1t2). This process is called a merge operation; repeat this process until there are no character pairs that can be merged or the number of merge operations reaches a preset value, such as 10,000 times. Since the node type is much smaller than the node value, there is no need to encode the node type. The final byte pair encoded vocabulary includes all the initial characters and the merged character pairs and the mergeable pairs.

[0037] Step S2: Code conversion phase: Use the extended attention mechanism to extract the structural features and sequence features of the code.

[0038] Step S2.1: Perform conversion processing based on the abstract syntax tree form obtained in step S1 to extract the structural features of the abstract syntax tree; examine each pair of nodes on the tree and find a unique path by traversing, and all paths together represent the relative structural information and hierarchical information of the abstract syntax tree; the path representation is as follows:

[0039] UD p ath(a, b)=U i D j

[0040] Among them U i represents the number of upward movements from node a to node b, D j Indicates the number of downward movements from node a to node b;

[0041] Introducing matrix R to represent UD in neural network calculation path The matrix formed is:

[0042]

[0043] Where Q, K, and V represent the matrix consisting of the query, key, and value vectors of the input elements. The softmax function is used to calculate the weight that determines the degree of expression of V. In order to avoid the QK product from being too large and causing the gradient to disappear, the result is divided by Alleviate the vanishing gradient problem. are the dimensions along which query, key, and value vectors are created for the input elements.

[0044] Step S2.2: After pre-order traversal, the abstract syntax tree sequence is obtained, and the relative position relationship of each pair of nodes in the sequence is examined to extract sequence features. The sequence features of the code reflect the developer's design logic, showing repeated patterns and regularity information in the sequence, which is helpful for understanding the semantics of the code. The extracted sequence features are the relative position encoding of the nodes in the sequence, which are specifically expressed as follows:

[0045]

[0046]

[0047] where z i is x i The attention score, W V , W Q , W K is a trainable matrix, a ij is x i For the current position x j The attention are two trainable parameters of the extended attention mechanism; a ij The calculation of is to get the probability distribution through the softmax function.

[0048] Step S3, training part: adopt Transformer decoder as the basic network model and extend the attention mechanism. The extension content is to introduce a new value R when calculating the query Q and the key K. R represents the relative hierarchical traversal path between nodes to capture the relative structural information; introduce a new variable and When calculating the attention score of a node, the score weight with another node is taken into account to extract the relative position relationship of the code sequence; finally, the probability is calculated through the softmax layer and the result is output.

[0049] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A code completion method based on structural features and sequence features, characterized in that: The following steps are involved: Step S1, code preprocessing; Collect source code from open source platforms and convert the source code into abstract syntax tree form; A byte pair encoding method is used to establish a corresponding vocabulary; Step S2, code conversion; Use the extended attention mechanism to extract the structural and sequential features of the code; Step S3, training part: using the Transformer decoder as the basic network model and extending the attention mechanism, wherein the extension content is to introduce new matrices and variables to capture the relative structural information and relative position relationship of the code respectively; finally, the probability is calculated through the softmax layer and the result is output; The specific steps of code conversion in step S2 include: Step S2.1, converting the abstract syntax tree obtained in step S1 to extract the structural features of the abstract syntax tree; examining each pair of nodes on the tree, finding a unique path by traversing, and all the paths together represent the relative structural information and hierarchical information of the abstract syntax tree; Step S2.2, after pre-order traversal, an abstract syntax tree sequence is obtained, the relative position relationship of each pair of nodes in the sequence is examined, and sequence features are extracted; The path representation in step S2.1 is as follows: UD path (a,b)=U i D j Among them U i represents the number of upward movements from node a to node b, D j Indicates the number of downward movements from node a to node b; Introducing matrix R to represent UD in neural network calculation path The matrix formed is: Where Q, K, V represent the matrix consisting of the query, key and value vectors of the input elements; the softmax function is used to calculate the weight that determines the degree of expression of V; are the dimensions for creating query, key, and value vectors for the input elements; The specific method for extracting sequence features in step S2.2 is as follows: The characteristics of code sequences are expressed as repeated patterns and regularity information. The extracted sequence features are the relative position encodings of the nodes on the sequence, which are specifically expressed as follows: where z i is x i The attention score, W V , W Q , W K is a trainable matrix, a ij is x i For the current position x j The attention are two trainable parameters of the extended attention mechanism; a ij The calculation of is to get the probability distribution through the softmax function.

2. A code completion method based on structural features and sequence features according to claim 1, characterized in that: In the step S1, the source code is converted into an abstract syntax tree form; each node on the tree includes two parts: a node type and a node value; when a node on a tree has no node value, it is filled with a placeholder [empty]; a vocabulary is established for the node type and the node value respectively, wherein a byte pair encoding method is used to construct the vocabulary for the node value.

3. A code completion method based on structural features and sequence features according to claim 2, characterized in that: The specific method of constructing a vocabulary for node values ​​using byte pair encoding in step S1 is as follows: The byte pair encoding algorithm is an iterative operation: first, each word in the corpus is segmented, and the single characters are added to the initial vocabulary; then the frequency of two consecutive characters is counted, and the character pair with the highest frequency (t1, t2) is merged into a new character (t1t2); this operation is repeated until there are no character pairs that can be merged or the number of merging operations reaches a preset value; finally, the vocabulary after byte pair encoding includes all the initial characters, merged character pairs, and mergeable pairs.