A code snippet recommendation method and system based on abstract syntax tree

Abstract

The present invention discloses a code snippet recommendation method and system based on an abstract syntax tree, including: parsing candidate codes into an abstract syntax tree, mapping each node therein into a hash value, counting the hash values ​​of all subtrees and the number of nodes contained therein, and recording the information of all subtrees in a list form; performing abstract syntax tree parsing, hash value mapping and subtree information statistics on the problem code in the same manner, and recording the hash value and the number of nodes of the corresponding subtree through a tree structure; performing node matching between the intermediate structures converted from the problem code and the candidate code, recording the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code as similarity and sorting, and presenting a sorted list to the user. The present invention converts the matching process of the abstract syntax tree into a matching of a tree and a list, greatly reducing the matching time while retaining the code structure information, and can be applied to recommendation scenarios with high requirements for immediacy, thereby improving the recommendation effect.

Description

Technical Field

[0001] The present invention relates to code recommendation, and in particular to a code snippet recommendation method and system based on an abstract syntax tree. Background Art

[0002] A code snippet is a set of code sequences that can guide developers to quickly master the usage of a program interface or the implementation of a programming task. In recent years, research in the field of code recommendation has focused on how to use information retrieval or machine learning to recommend code. Most methods return code snippets for a given problem, while there are still few methods that use programming context as input for recommendation. In addition, since the recommendation task has high requirements for immediacy, these methods often use simple text information as code representation, use sequence or bag-of-words methods for matching, lack grammatical analysis, and ignore the structural information of the code.

[0003] The abstract syntax tree is an abstract representation of the syntax structure of the source code. It represents the syntax structure of the programming language in a tree-like form, and each node on the tree represents a structure in the source code. The reason why the syntax is "abstract" is that the syntax here does not represent every detail that appears in the real syntax. Therefore, modifications to the code at the lexical level will not lead to changes in the abstract syntax tree. Although using the abstract syntax tree for matching can achieve better results, it involves subtree matching problems and takes a long time, so it is not suitable for code recommendation scenarios. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a code snippet recommendation method and system based on an abstract syntax tree, which avoids the huge time overhead of the traditional abstract syntax tree matching method and is suitable for recommendation scenarios with high requirements for immediacy, thereby improving the recommendation effect.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0006] A code snippet recommendation method based on an abstract syntax tree comprises the following steps:

[0007] (1) Parse the candidate code in the code database into an abstract syntax tree, map each node into a hash value, count the hash values ​​of all subtrees in the abstract syntax tree and the number of nodes contained therein, and record the information of all subtrees in the candidate code in a list form; the hash value of a subtree is the sum of the hash values ​​of each node in the subtree;

[0008] (2) The problem code is also parsed through an abstract syntax tree, hash value mapping, and subtree information statistics. The hash value and node number information of the corresponding subtree are recorded in a tree structure. Each node records the hash value and node number of the subtree with the node as the root in the abstract syntax tree.

[0009] (3) Perform a pre-order traversal on the tree structure converted from the problem code, match the hash values ​​of the subtrees with the same number of nodes in the candidate code, record the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code, sort them according to the number of repeated nodes, and present the sorted list to the user.

[0010] Preferably, in step (1), nodes with similar functions are mapped to the same hash value.

[0011] Preferably, in step (1), the candidate code information is recorded in the following manner:

[0012] Represent each subtree in the abstract syntax tree as a two-tuple of <number of nodes, hash value>. Duplicate two-tuples are recorded only once. The hash values ​​of subtrees with the same number of nodes are sorted and recorded in a two-dimensional list. The first dimension of the two-dimensional list represents the number of nodes, and the second dimension represents the hash value list of all subtrees corresponding to the number.

[0013] Preferably, the index of the code and the total number of nodes are recorded at the position where the number of nodes in the two-dimensional list is 0.

[0014] Preferably, in step (3), code matching is performed in the following manner:

[0015] Pre-order traversal is used to convert the problem code into a tree structure, obtain a list of subtree hash values ​​with the same number of nodes in the candidate code, and search whether there is a node in the candidate list with the same hash value as the current node; if a hit is found, the number of subtree nodes is recorded, and all subtree nodes of the node are skipped in subsequent matching; after the pre-order traversal is completed, the sum of the number of nodes of all identical subtrees is recorded as the similarity score.

[0016] Preferably, the hash value list of subtrees with the same number of nodes in the candidate code is ordered, and a binary search is performed to find out whether there is a node with the same hash value as the current node in the candidate list.

[0017] Preferably, before matching with the candidate code, it is first determined whether a set ratio of the number of nodes contained in the problem code tree is greater than the number of nodes in the candidate code. If so, subsequent matching is performed, otherwise matching is skipped directly; the ratio range is 70%-90%.

[0018] Preferably, when the number of hit nodes exceeds the set ratio of the total number of problem code nodes, the tree is additionally verified to avoid mismatching caused by different subtrees being mapped to the same hash value; the ratio range is 40%-50%, and the additional verification is to verify whether the 5 subsequent nodes of the node can also be found in the candidate list. If all can be found, it is deemed to have passed the verification.

[0019] Based on the same inventive concept, the present invention provides a code snippet recommendation system based on an abstract syntax tree, comprising:

[0020] The candidate code processing module is used to parse the candidate code in the code database into an abstract syntax tree, map each node therein to a hash value, count the hash values ​​of all subtrees in the abstract syntax tree and the number of nodes contained therein, and record the information of all subtrees in the candidate code in a list form; wherein the hash value of a subtree is the sum of the hash values ​​of each node in the subtree;

[0021] The problem code processing module is used to perform abstract syntax tree analysis, hash value mapping and subtree information statistics on the problem code, and record the hash value and node quantity information of the corresponding subtree through a tree structure; each node records the hash value and node quantity of the subtree with the node as the root in the abstract syntax tree;

[0022] And, the code recommendation module is used to perform a pre-order traversal on the tree structure converted from the problem code, match the hash values ​​of the subtrees with the same number of nodes in the candidate code, record the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code, sort them according to the number of repeated nodes, and present the sorted list to the user.

[0023] Based on the same inventive concept, the present invention provides a code snippet recommendation system based on an abstract syntax tree, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the code snippet recommendation method based on the abstract syntax tree when loaded into the processor.

[0024] Beneficial effects: Traditional query-free code recommendation methods are often matched based on code text information and lack mining of code structure information. Using the code abstract syntax tree for matching can take into account the code syntax information and achieve better results, but it will take a lot of time and is not suitable for code recommendation scenarios. The present invention extracts information from the abstract syntax tree, converts the candidate code into a list structure, retains the tree structure of the problem code, and converts the tree matching problem into a tree-list match, avoiding the huge time overhead of traditional tree matching methods and adapting them to the need for immediacy in code recommendation tasks; the similarity score of the candidate method is obtained by the maximum similarity subtree matching method, and a recommendation list is obtained based on this, so that the method is capable of recommending candidate codes that are lexically different from the problem code but have similar grammatical structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of an abstract syntax tree and node hash values ​​corresponding to an example code in an embodiment of the present invention.

[0027] Figure 3 It is a schematic diagram of the intermediate representation structure of the example code in the embodiment of the present invention when it is used as the question code.

[0028] Figure 4 It is a schematic diagram of the intermediate representation structure of the example code when it is used as a candidate code in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0030] like Figure 1 As shown, a code snippet recommendation method based on an abstract syntax tree disclosed in an embodiment of the present invention mainly involves two parts, a data processing stage and a matching stage.

[0031] In the data processing stage, there are two situations: the current context code (i.e., the query code as a question, referred to as the question code in this invention) and the candidate code in the code database. The processing steps of the two are similar, and they only differ in the final storage structure. The processing process is mainly divided into the following three steps:

[0032] 1) The parser converts the code to be processed into an abstract syntax tree. Currently, we have implemented the method in Java and C languages, using javalang to parse Java code and Clang to parse C language code. In addition, for individual Java methods, we have added foo_class so that it can be processed by the parser.

[0033] 2) Map each node in the abstract syntax tree to a hash value. In principle, different abstract syntax tree nodes should have different hash values, but in order to improve the detection effect, we map nodes with similar functions to the same hash value to avoid the reduction of similarity due to the replacement of similar nodes. For example, for the following code segment:

[0034] for(int i=0;i<10;i++)

[0035] {

[0036] a+=0.5;

[0037] }

[0038] Its abstract syntax tree and the hash value of each node are as follows Figure 2 As shown in the figure, since integer type and floating point type are both numerical types, the same hash value 17 is assigned to these two numerical types. In addition, nodes with similar functions also include similar loops and judgments, such as for statement nodes and while statement nodes that both represent loops, and if statement nodes and switch statement nodes that both represent judgments. In specific implementation, supplements or adjustments can be made based on the detection results.

[0039] 3) Calculate the hash values ​​of all subtrees in the abstract syntax tree. We simply define the hash value of a tree as the sum of the hash values ​​of each node in the tree, and record the total number of nodes contained in each tree. For the current context code used as a query, we retain the tree structure consistent with its abstract syntax tree, and record the number of nodes and hash values ​​of the subtree with the current node as the root in the original abstract syntax tree at each node position. Figure 3 is the intermediate tree structure converted from the code in 2). For the candidate codes, we discard the tree structure and the number of nodes (that is, we only record the same subtree once) and store them only in a list form, such as Figure 4 As shown. For each candidate code, there is a corresponding candidate code list. The first dimension of the list represents the number of nodes, and the second dimension is the hash value list formed by all subtrees corresponding to the number. The hash value list corresponding to each node number will be kept in order to speed up the subsequent search process. In addition, we also record the index of the code and the total number of nodes at the position where the number of nodes in each candidate code list is 0.

[0040] In the matching stage, we will calculate the total number of nodes in all common subtrees between the problem code and the candidate code and use it as the code similarity. This stage is divided into the following two parts:

[0041] 1) Determine whether 80% of the number of nodes in the problem code tree is greater than the number of nodes in the candidate code. If so, perform subsequent matching, otherwise skip matching directly and set the similarity to 0. For candidate codes with fewer nodes than the problem code, the maximum similarity score is the number of candidate code nodes / the number of problem code nodes. After experimental verification, we found that the similarities of the top ten in the recommendation result list are often above 80%, so we choose to filter out these candidate codes with smaller node numbers to reduce matching time. In addition, in the code recommendation task, developers often need to use unfinished code as a query and obtain complete code as a reference. Therefore, generally speaking, the recommended results should also be greater than the context code as the problem. In order to avoid incorrectly filtering out similar codes, for the sake of conservatism, in this embodiment, we set the node number threshold to 80%. Of course, the threshold can also be adjusted appropriately according to demand, such as setting it within the range of 70%-90%.

[0042] 2) Pre-order traversal of the tree-type intermediate representation of the problem code, and search the candidate code for a list of hash values ​​with the same value as the number of nodes currently visited. Since the hash value list is ordered, a binary search method is used for searching. If the same hash value as the current node is found in the list, all child nodes of the node are skipped in subsequent traversals. In addition, since only one number is used to represent the hash value, when the code base is large, the abstract syntax tree containing different nodes will be mapped to the same hash value. Since the probability of this event occurring is not large, only the mismatch of the larger code tree will affect the result, while the impact of the small code tree can be ignored. Therefore, this embodiment performs additional verification on the tree when the number of hit nodes exceeds 40% of the total number of problem code nodes (the threshold can be adjusted as needed). We have adopted a simple but effective method, that is, verifying whether the 5 nodes after the node can also be found in the candidate list. If all can, it is considered to have passed the verification. After the traversal is completed, the sum of the number of all hit nodes is counted, the similarity is calculated, and the recommendation list is returned after sorting.

[0043] Based on the same inventive concept, an embodiment of the present invention provides a code snippet recommendation system based on an abstract syntax tree, including: a candidate code processing module, which is used to parse the candidate code in the code database into an abstract syntax tree, map each node therein to a hash value, count the hash values ​​of all subtrees in the abstract syntax tree and the number of nodes contained, and record the information of all subtrees in the candidate code in a list form; wherein the hash value of a subtree is the sum of the hash values ​​of each node in the subtree; a problem code processing module, which is used to perform abstract syntax tree parsing, hash value mapping and subtree information statistics on the problem code, and record the hash value and node number information of the corresponding subtree through a tree structure; wherein each node records the hash value and node number of the subtree with the node as the root in the abstract syntax tree; and a code recommendation module, which is used to perform a pre-order traversal on the tree structure converted from the problem code, match the hash values ​​of the subtrees with the same number of nodes in the candidate code, record the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code, sort them according to the number of repeated nodes, and present a sorted list to the user. For specific implementation details, please refer to the above method embodiment, which will not be repeated here.

[0044] Based on the same inventive concept, an embodiment of the present invention provides a code snippet recommendation system based on an abstract syntax tree, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned code snippet recommendation method based on the abstract syntax tree when loaded into the processor.

Claims

1. A code snippet recommendation method based on an abstract syntax tree, characterized in that: The steps include: (1) Parse the candidate code in the code database into an abstract syntax tree, map each node into a hash value, count the hash values ​​and the number of nodes contained in all subtrees in the abstract syntax tree, and record the information of all subtrees in the candidate code in a list form; The hash value of a subtree is the sum of the hash values ​​of each node in the subtree; the hash values ​​of subtrees with the same number of nodes are sorted; (2) Perform abstract syntax tree analysis, hash value mapping, and subtree information statistics on the problem code, and record the hash value and node number information of the corresponding subtree through a tree structure; Each node records the hash value and number of nodes in the subtree with the node as the root in the abstract syntax tree; (3) Perform a pre-order traversal on the tree structure converted from the problem code, match the hash values ​​of the subtrees with the same number of nodes in the candidate code, record the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code, sort them according to the number of repeated nodes, and present the sorted list to the user.

2. The code snippet recommendation method based on abstract syntax tree according to claim 1, characterized in that: In the step (1), nodes with similar functions are mapped to the same hash value.

3. The code snippet recommendation method based on abstract syntax tree according to claim 1, characterized in that: In step (1), the candidate code information is recorded in the following manner: Each subtree in the abstract syntax tree is represented as a two-tuple of <number of nodes, hash value>. Duplicate two-tuples are recorded only once in a two-dimensional list. The first dimension of the two-dimensional list represents the number of nodes, and the second dimension represents the hash value list of all subtrees corresponding to the number.

4. The code snippet recommendation method based on abstract syntax tree according to claim 3, characterized in that: The index of the code and the total number of nodes are recorded at the position where the number of nodes in the two-dimensional list is 0.

5. The code snippet recommendation method based on abstract syntax tree according to claim 1, characterized in that: In step (3), code matching is performed in the following manner: Pre-order traverse the tree structure converted from the problem code, obtain the hash value list of the subtrees with the same number of nodes in the candidate code, and find out whether there is a node in the candidate list with the same hash value as the current node; If a hit is found, the number of nodes in the subtree is recorded, and all subtree nodes of the node are skipped in subsequent matches; after the pre-order traversal is completed, the sum of the number of nodes in all identical subtrees is recorded as the similarity score.

6. The code snippet recommendation method based on abstract syntax tree according to claim 5, characterized in that: The list of subtree hash values ​​with the same number of nodes in the candidate code is ordered, and a binary search is performed to find out whether there is a node in the candidate list with the same hash value as the current node.

7. The code snippet recommendation method based on abstract syntax tree according to claim 5, characterized in that: Before matching with the candidate code, first determine whether the set ratio of the number of nodes contained in the problem code tree is greater than the number of nodes in the candidate code. If it is satisfied, subsequent matching is performed, otherwise the matching is skipped directly; the ratio range is 70%-90%.

8. The code snippet recommendation method based on abstract syntax tree according to claim 5, characterized in that: When the number of hit nodes exceeds the set ratio of the total number of problem code nodes, the tree is additionally verified to avoid mismatching caused by different subtrees being mapped to the same hash value; the ratio range is 40%-50%, and the additional verification is to verify whether the 5 subsequent nodes of the node can also be found in the candidate list. If they can all be found, it is considered to have passed the verification.

9. A code snippet recommendation system based on an abstract syntax tree, characterized in that: include: The candidate code processing module is used to parse the candidate code in the code database into an abstract syntax tree, map each node therein to a hash value, count the hash values ​​and the number of nodes contained in all subtrees in the abstract syntax tree, and record the information of all subtrees in the candidate code in a list form; The hash value of a subtree is the sum of the hash values ​​of each node in the subtree; the hash values ​​of subtrees with the same number of nodes are sorted; The problem code processing module is used to perform abstract syntax tree analysis, hash value mapping and subtree information statistics on the problem code, and record the hash value and node quantity information of the corresponding subtree through a tree structure; Each node records the hash value and number of nodes in the subtree with the node as the root in the abstract syntax tree; And, the code recommendation module is used to perform a pre-order traversal on the tree structure converted from the problem code, match the hash values ​​of the subtrees with the same number of nodes in the candidate code, record the total number of nodes contained in all the largest repeated subtrees of each candidate code and the problem code, sort them according to the number of repeated nodes, and present the sorted list to the user.

10. A code snippet recommendation system based on an abstract syntax tree, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the method for recommending code snippets based on an abstract syntax tree according to any one of claims 1 to 8 is implemented.

Images