Code semantic block method and system based on hierarchical tree and graph community

Abstract

The application provides a code semantic segmentation method and system based on a hierarchical tree and a graph community, and relates to the technical field of code analysis. A code hierarchical tree is constructed according to the indentation level of each line of code in the source code, which can effectively capture the structured information of the code and has a lower requirement for the syntax integrity of the code. The code hierarchical tree is converted into an undirected graph. A community discovery algorithm is used to identify the undirected graph and divide the undirected graph into multiple communities, which can identify the code communities in the graph that are closely connected and have stronger semantic cohesion. The multiple communities are optimized to obtain the code semantic segmentation result, ensuring the semantic integrity, continuity and appropriate size of the final segmentation result. More accurate and more context-related code segmentation can be provided, and the performance of the code retrieval enhancement generation system and the quality of the generated code can be significantly improved.

Description

Technical Field

[0001] This invention relates to the field of code analysis technology, and more specifically, to a code semantic segmentation method and system based on hierarchical trees and graph communities. Background Technology

[0002] In Retrieval Augmentation (RAG) systems, chunking is the process of breaking down large documents or codebases into smaller, more manageable pieces that can then be retrieved and used as context for the generative model. Effective code chunking is crucial for code-based RAG systems because it directly impacts the accuracy of retrieval and the quality of generated code. Inappropriate chunking strategies can lead to the retrieval of incomplete or semantically incoherent code fragments, misleading the model into generating incorrect or irrelevant code. For example, if the chunking process arbitrarily cuts off function definitions or class declarations, the retrieved fragments will lack the necessary context, making it difficult for the model to understand their functionality or integrate them correctly. Therefore, designing chunking strategies that preserve the logical structure and semantic integrity of the code is a core element in building efficient code-based RAG systems.

[0003] In existing methods, such as Figure 1 As shown, using an Abstract Syntax Tree (AST) for structure-aware chunking relies on precise AST parsing. This method will fail directly for code with incomplete syntax (such as code snippets), code containing non-standard syntax, or code with specific syntax not supported by the parser. Chunk boundaries are strictly defined by the AST's syntax nodes (such as functions and classes), making it difficult to flexibly capture higher-level logical units or functional modules that span multiple syntactic structures. Summary of the Invention

[0004] The problem this invention aims to solve is that existing AST segmentation methods are highly dependent on the integrity of code parsing and lack sufficient segmentation flexibility.

[0005] To address the aforementioned problems, in a first aspect, this invention provides a code semantic segmentation method based on hierarchical trees and graph communities, comprising: Construct a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree. Lines of code with the same indentation level belong to the same indentation level. Each line of code is a node in a code hierarchy tree. Convert the code hierarchy tree into an undirected graph; A community detection algorithm is used to identify undirected graphs and segment them into multiple communities. Optimize multiple communities to obtain the code semantic block results.

[0006] Optionally, converting the code hierarchy tree into an undirected graph includes: In the code hierarchy tree, nodes are encoded, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code. Remove the root node, keep the parent-child edges, and add sibling edges between nodes in the same level according to their node numbers from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the connection between two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the connection between nodes in the same level in the code hierarchy tree. In the initial undirected graph, sequential edges are added in ascending order of node numbers to connect all nodes and obtain the final undirected graph.

[0007] Optionally, the optimization process for multiple communities to obtain the code semantic segmentation result includes: If the node numbers within a community are not consecutive, then the connection between two nodes with non-consecutive node numbers in the community will be broken, resulting in multiple sub-communities with consecutive node numbers. If the space occupied by the club is greater than the first preset space, the club will be divided into multiple sub-clubs, and the space occupied by each sub-club will be less than the preset space. A greedy strategy is adopted to greedily merge sub-communities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent, resulting in multiple final communities. The source code is then divided according to the node numbers within the final community to obtain the code semantic block result.

[0008] Optionally, the step of dividing the community into multiple sub-communities if the community's occupied space is greater than the first preset occupied space, and where the occupied space of each of the multiple sub-communities is less than the preset occupied space, includes: If the space occupied by a community is greater than the first preset space, it is recorded as a community to be split. The community to be split is split with the last non-leaf node as the dividing point to obtain two sub-communities. If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. The splitting continues with the last non-leaf node in the sub-community to be split as the dividing point until the space occupied by the resulting sub-communities is less than the preset space.

[0009] Secondly, the present invention also provides a code semantic segmentation system based on hierarchical trees and graph communities, comprising: The hierarchical tree building module is used to build a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree, lines of code with the same indentation amount belong to the same indentation level, and each line of code is a node in a code hierarchy tree. The hierarchy tree conversion module is used to convert a code hierarchy tree into an undirected graph; The undirected graph segmentation module is used to identify undirected graphs using a community detection algorithm and segment the undirected graph into multiple communities. The community optimization module is used to optimize multiple communities and obtain code semantic block results.

[0010] Optionally, the hierarchical tree conversion module includes: The encoding submodule is used to encode nodes in the code hierarchy tree, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code. The sibling edge is added as a submodule to remove the root node and retain the parent-child edge. Sibling edges are added between nodes in the same level according to their node numbers from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the edge connecting two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the edge connecting nodes in the same level in the code hierarchy tree. The Sequential Edge Addition submodule is used to add sequential edges in the initial undirected graph in ascending order of node number, connecting all nodes to obtain the final undirected graph.

[0011] Optionally, the community optimization module includes: The first optimization submodule is used to disconnect the connection between two nodes with non-continuous node numbers in a community if the node numbers in the community are not consecutive, so as to obtain multiple sub-communities with consecutive node numbers. The second optimization submodule is used to divide the community into multiple sub-communities if the community's occupied space is greater than the first preset occupied space, and the occupied space of each of the multiple sub-communities is less than the preset occupied space. The greedy merging submodule is used to employ a greedy strategy to merge sub-communities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent, resulting in multiple final communities. The source code is then divided according to the node numbers within the final community to obtain the semantic code segmentation result.

[0012] Optionally, the second optimization submodule includes: If the space occupied by a community is greater than the first preset space, it is recorded as a community to be split. The community to be split is split with the last non-leaf node as the dividing point to obtain two sub-communities. If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. The splitting continues with the last non-leaf node in the sub-community to be split as the dividing point until the space occupied by the resulting sub-communities is less than the preset space.

[0013] Thirdly, the present invention provides an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the code semantic segmentation method based on hierarchical trees and graph communities as described in the first aspect when executing the computer program.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the code semantic segmentation method based on hierarchical trees and graph communities as described in the first aspect.

[0015] This invention provides a code semantic segmentation method and system based on hierarchical trees and graph communities. Compared with existing technologies, it has the following advantages: By constructing a code hierarchy tree, this method effectively captures the structured information of the code. It has low requirements for syntactic integrity, and can represent even incomplete or non-standard code snippets structurally based on indentation levels. Converting the hierarchy tree into an undirected graph and applying a community detection algorithm allows the chunking process to move beyond strict syntactic node boundaries and identify tightly connected, semantically cohesive code communities. This enables the method to flexibly capture high-level logical units or functional modules that span multiple syntactic structures. Finally, multi-stage optimization of the communities further ensures the semantic integrity, continuity, and appropriate size of the final chunking results, thus avoiding the problem of retrieving incomplete or semantically incoherent code snippets that may occur with existing methods. This provides more accurate and context-sensitive code chunking, significantly improving the performance of the code retrieval enhancement generation system and the quality of the generated code. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating existing code segmentation provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a code semantic segmentation method based on hierarchical trees and graph communities provided in an embodiment of the present invention; Figure 3 A flowchart illustrating a code semantic block segmentation method provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the construction of a code hierarchy tree provided in an embodiment of the present invention; Figure 5This is a schematic diagram illustrating the construction process of an undirected graph according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a community optimization process provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a code semantic block system based on hierarchical trees and graph communities provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0020] like Figure 2 As shown in the embodiment of this application, a code semantic segmentation method based on hierarchical trees and graph communities is provided, including: S1: Construct a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree. Lines of code with the same indentation level belong to the same indentation level. Each line of code is a node in a code hierarchy tree.

[0021] S2: Convert the code hierarchy tree into an undirected graph.

[0022] S3: Use a community detection algorithm to identify undirected graphs and divide them into multiple communities.

[0023] S4: Optimize multiple communities to obtain the code semantic block results.

[0024] In this optional embodiment, by constructing a code hierarchy tree, the structured information of the code can be effectively captured. This differs from the existing method that relies on Abstract Syntax Trees (ASTs) for parsing. The method presented in this application has lower requirements for the syntactic integrity of the code, and can perform structured representation based on indentation levels even when faced with syntactically incomplete or non-standard code fragments. Secondly, by converting the hierarchy tree into an undirected graph and applying a community detection algorithm, the chunking process is no longer limited to strict syntactic node boundaries, but can identify tightly connected and more semantically cohesive code communities in the graph. This allows the method to flexibly capture high-level logical units or functional modules that span multiple syntactic structures, overcoming the lack of flexibility in existing AST chunking methods. Finally, through multi-stage optimization of the communities, the semantic integrity, continuity, and appropriate size of the final chunking result are further ensured, thereby avoiding the problem of retrieving incomplete or semantically incoherent code fragments that may be caused by existing methods. Thus, this method can provide more accurate and more context-sensitive code chunking, significantly improving the performance of the code retrieval augmented generation (RAG) system and the quality of the generated code.

[0025] The following is combined with Figures 3-6 Each step is described in detail.

[0026] S1: Construct a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree. Lines of code with the same indentation level belong to the same indentation level. Each line of code is a node in a code hierarchy tree.

[0027] Specifically, such as Figure 4 As shown, indentation level refers to the depth of indentation of each line of code in the source code relative to the beginning of its code block. It is usually measured by the number of leading spaces or tabs. Different indentation levels reflect the nesting level of the code and are the basis for building the code hierarchy tree.

[0028] In this process, each indentation level corresponds to a level in the code hierarchy tree. Lines of code with the same indentation level belong to the same indentation level, and each line of code is considered a node in the code hierarchy tree. For example, for a piece of Python code, its indentation level can be determined by counting the number of spaces or tabs at the beginning of each line of code. If a line of code has 4 spaces indentation, and the next line has 8 spaces indentation, the latter is considered a child node of the former. As another implementation, the indentation level can be determined by combining indentation characters and block delimiters (such as curly braces) according to the syntax rules of a specific programming language. For example, in C++ or Java code, even if the code inside curly braces has the same indentation as the outer code, it is logically at a deeper level. In this case, its level in the hierarchy tree can be adjusted according to the nesting depth of the curly braces. This method has lower requirements for the syntactic correctness of the code and only relies on formatting information.

[0029] S2: Convert the code hierarchy tree into an undirected graph.

[0030] Specifically, one implementation is to treat each node in the hierarchy tree (i.e., each line of code) as a vertex in an undirected graph. For parent-child relationships existing in the hierarchy tree, an edge is added to the undirected graph connecting the parent and child nodes. For example, if line A is the parent of line B, an edge connecting A and B is added to the undirected graph. Another implementation, besides parent-child relationships, can also consider adding edges between adjacent nodes at the same level in the hierarchy tree. For example, if lines C and D are at the same indentation level and are consecutive at that level, an edge connecting C and D is added to the undirected graph (e.g., ...). Figure 3 The connection between nodes 3 and 6 in the undirected graph reflects their sequential relationship. Simply converting the hierarchical tree structure into a graph may not fully capture important information such as the physical order of the code and the logical connections between statements at the same level, resulting in a limited semantic representation of the generated undirected graph and affecting the accuracy of the final code semantic segmentation. Therefore, this step needs to be further designed, such as... Figure 5 As shown, the process of converting the code hierarchy tree into an undirected graph includes: S210: Encode nodes in the code hierarchy tree, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code.

[0031] Specifically, such as Figure 5As shown, each line of code (node) is assigned a unique, sortable identifier (i.e., a number on the node) so that structured connections can be established based on these identifiers during subsequent graph construction. Setting the root node as a virtual node and encoding it as zero allows for unified processing of all top-level code blocks and ensures that all actual line of code nodes have positive line number encoding, facilitating management and operation. Node numbers are line numbers, directly utilizing the inherent order information of the source code, giving the encoding intuitive semantics and reflecting the physical location of the code. This encoding process can be achieved by traversing the code hierarchy tree and assigning the line number in the source code file to each non-root node (i.e., actual line of code) as its encoding; or by storing the line number of each line of code as an attribute of its corresponding node while constructing the code hierarchy tree during source code parsing.

[0032] S220: Remove the root node, keep the parent-child edges, and add sibling edges between nodes in the same level according to the node number from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the connection edge between two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the connection edge between nodes in the same level in the code hierarchy tree.

[0033] Specifically, when constructing the initial undirected graph, virtual root nodes are removed, focusing on the relationships between actual lines of code to avoid interference with the semantic analysis of the actual code. For example... Figure 5 As shown, parent-child edges are preserved (i.e. Figure 5 Adding parent-child edges (to the code) can reflect the nesting structure and hierarchical relationship of the code, such as the containment relationship between function definitions and function bodies, and loop statements and loop bodies. Within the same level, sibling edges (i.e., lines of code) are added in order of node number (i.e., line number). Figure 5 Add sibling edges. These sibling edges connect logically adjacent lines of code within the same code block, such as consecutive statements within the same function body; they often work together to complete a sub-function. By combining parent-child edges and sibling edges, the initial undirected graph can effectively capture the structured semantics and local relationships of the code.

[0034] S230: In the initial undirected graph, add sequential edges in ascending order of node numbers to connect all nodes and obtain the final undirected graph.

[0035] Specifically, to further enhance the connectivity of the graph and reflect the overall execution flow of the code, this application adds sequential edges (i.e., ...) in ascending order of node numbers to the initial undirected graph. Figure 5Adding line number edges connects all nodes. These sequential edges connect physically adjacent lines of code in the source code, even if they are not directly adjacent in the hierarchical tree, because they are continuous in execution flow. This global sequential connection ensures that the community detection algorithm can comprehensively consider the code structure, local semantics, and overall linear execution order when identifying code semantic blocks. Through the construction of this multi-dimensional edge, the final undirected graph can more comprehensively and accurately represent the semantic relationships of the source code, thus providing high-quality input for subsequent community detection algorithms, enabling the segmented communities to more accurately correspond to the actual code semantic blocks.

[0036] S3: Use a community detection algorithm to identify undirected graphs and divide them into multiple communities.

[0037] Specifically, such as Figure 3 As shown, algorithms based on modularity optimization, such as the Louvain algorithm (a community detection algorithm), can be used. This algorithm can automatically identify closely connected groups of nodes in the graph, i.e., "communities." These communities often correspond to highly cohesive logical units in the code, and their boundaries may naturally cross traditional AST syntax nodes. The algorithm iteratively assigns nodes to communities to maximize the density of connections within communities while minimizing the density of connections between communities. For example, the Louvain algorithm first treats each node as an independent community, and then gradually merges closely connected nodes into larger communities through local optimization and community aggregation steps. Alternatively, the Label Propagation Algorithm (LPA) can be used. This algorithm identifies communities by propagating labels in the graph. Each node updates its own label based on the labels of its neighbors until the labels of all nodes converge, thus forming a stable community structure.

[0038] S4: Optimize multiple communities to obtain the code semantic block results.

[0039] Specifically, such as Figure 6 As shown, simple merging or splitting operations can be performed on communities. For example, if two adjacent communities are consecutive in the code and are considered to have high functional relevance through manual judgment or heuristic rules, they can be merged. If a community is deemed too large, it can be split into smaller parts, for example, based on blank lines or comment blocks in the code. The specific implementation process of step S4 is as follows: S410: If the node numbers within a community are not consecutive, then disconnect the connection between two nodes with non-consecutive node numbers in the community to obtain multiple sub-communities with consecutive node numbers.

[0040] Specifically, such as Figure 6As shown, when node numbers within a community are discontinuous, it indicates that the community may contain lines of code that are physically discontinuous in the source code. This usually means that these lines of code do not belong to the same semantic unit. To ensure the semantic integrity and physical continuity of each code block, the connection between two nodes with discontinuous node numbers within a community is broken, thus splitting a community into multiple sub-communities with consecutive node numbers. This can be done by traversing all nodes within a community and checking whether their corresponding line numbers are consecutive. Once a jump in line numbers is found, the community is logically divided based on this boundary. Alternatively, all consecutive node sequences within a community can be identified, and each consecutive sequence can be treated as an independent sub-community. For example... Figure 6 In the middle, the numbering of community A is not continuous, so it is broken between node 5 and node 7 to obtain two sub-communities.

[0041] S420: If the space occupied by the community is greater than the first preset space, the community will be divided into multiple sub-communities, and the space occupied by each of the multiple sub-communities will be less than the preset space.

[0042] Specifically, such as Figure 6 As shown, the community's occupied space is the resource amount occupied by all the lines of code constituting that community. This can be measured by various metrics, such as the number of lines of code, the number of characters, the number of lexical units (tokens), or the number of nodes. An excessively large community may encompass multiple independent semantic functions, hindering fine-grained code understanding and analysis. Therefore, when the community's occupied space (e.g., the number of lines of code, the number of characters, or the number of abstract syntax tree nodes) exceeds a first preset occupied space, the community will be divided into multiple sub-communities. The purpose of this division is to ensure that the occupied space of each sub-community is less than the preset occupied space, thereby guaranteeing a suitable granularity of the code blocks. The division operation can be based on the structural characteristics of the code, for example, splitting at the boundaries of function definitions, class definitions, or major control structures (such as loops and conditional statements). Recursive division can also be used, continuously splitting the excessive sub-communities until the condition is met. Through division, it is ensured that the line numbers within each block are continuous and the size is controllable. This step is described in detail, S420 includes: S421: If the space occupied by a community is greater than the first preset space occupied, it is recorded as a community to be split. The last non-leaf node in the community to be split is used as the dividing point to split it into two sub-communities.

[0043] S422: If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. Continue to split the sub-community to be split using the last non-leaf node as the dividing point until the space occupied by the sub-communities is less than the preset space.

[0044] Specifically, the last non-leaf node refers to the last node with child nodes in the code hierarchy tree structure corresponding to the community to be split, arranged in line number order. For example... Figure 6 Node 6 in Club B.

[0045] To maintain the semantic integrity of the code as much as possible during segmentation, the last non-leaf node in the community to be segmented is chosen as the split point. In the code hierarchy tree, a non-leaf node typically represents a complete code structural unit, such as a function, a class, a loop body, or a conditional branch. Splitting using the last non-leaf node as the split point allows a large code block to be divided at a logically relatively complete position, resulting in two sub-communities. This segmentation method avoids arbitrary splitting in the middle of the code structure, helping to maintain the semantic coherence of the sub-communities after segmentation. Since the two sub-communities obtained from the initial split may still have a space occupied by more than a first preset space, the design is further improved so that if the space occupied by any sub-community still exceeds a preset threshold, it is regarded as a new sub-community to be segmented, and the same segmentation logic is continuously applied, that is, splitting it using the last non-leaf node in the sub-community to be segmented as the split point. This iterative process continues until the space occupied by all the finally formed sub-communities is less than or equal to the first preset space. This iterative, structured splitting mechanism ensures that all code semantic blocks retain their internal semantic integrity to the maximum extent while meeting size constraints, thus providing high-quality input for subsequent code analysis and processing.

[0046] S430: Using a greedy strategy, sub-communities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent are greedily merged to obtain multiple final communities. The source code is then divided according to the node numbers within the final community to obtain the semantic code segmentation result, such as... Figure 3 As shown, the source code text is divided into block 1 and block 2 according to the final node number within the community.

[0047] Specifically, such as Figure 6As shown, after the above segmentation process, some sub-communities may appear that occupy too little space but are logically adjacent. If these sub-communities exist alone, their semantics may not be complete. To improve the semantic completeness and cohesion of the code block, a greedy strategy is adopted to merge sub-communities whose space is less than or equal to the second preset space and whose node numbers are adjacent. Here, "greedy strategy" means that in each merge operation, adjacent sub-communities that can form a valid merge (i.e., still satisfy the second preset space limit after merging) are prioritized for merging. For example, starting from the beginning of the source code, adjacent sub-communities can be checked sequentially. If they are all small and the merged size will not exceed the preset size, they are merged until no more merges can be performed, such as... Figure 6 Community 1 and Community 2 are merged, and Community 3 and Community 4 are merged. A greedy merging strategy is adopted to greedily merge communities that are too small and semantically adjacent, in order to improve the information density of the blocks and make the block size distribution more balanced, thus ensuring the practicality of the final blocks and compatibility with the existing RAG process.

[0048] In this optional embodiment, by ensuring continuity first and then adjusting the granularity (breaking down large blocks into smaller ones first), the preliminary results of the community detection algorithm can be transformed into high-quality code semantic blocks. These blocks are not only physically continuous and of appropriate granularity, but also more semantically cohesive, thereby significantly improving the accuracy and efficiency of code analysis.

[0049] like Figure 7 As shown in the embodiment of this application, a code semantic segmentation system based on hierarchical trees and graph communities is provided, including: The hierarchical tree construction module 10 is used to construct a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree, and lines of code with the same indentation amount belong to the same indentation level. Each line of code is a node in a code hierarchy tree.

[0050] Hierarchical tree conversion module 20 is used to convert a code hierarchy tree into an undirected graph.

[0051] The undirected graph segmentation module 30 is used to identify undirected graphs using a community detection algorithm and segment the undirected graph into multiple communities.

[0052] The community optimization module 40 is used to optimize multiple communities and obtain code semantic block results.

[0053] In this embodiment, the system achieves a unity of structure awareness and semantic coherence during code segmentation. Compared to existing AST segmentation methods, this system is no longer limited by the completeness of syntax parsing and can adapt to a wider range of code scenarios. At the same time, through the graph community discovery mechanism, the segmentation results can naturally reflect the logical functional units of the code, significantly improving the accuracy of code retrieval and the quality of generated code, providing reliable technical support for building an efficient code retrieval enhancement generation system.

[0054] In an optional embodiment of this application, the hierarchical tree conversion module 20 includes: The encoding submodule is used to encode nodes in the code hierarchy tree, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code.

[0055] The sibling edge submodule is used to remove the root node and retain the parent-child edge. Sibling edges are added between nodes in the same level according to their node numbers from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the edge connecting two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the edge connecting nodes in the same level in the code hierarchy tree.

[0056] The Sequential Edge Addition submodule is used to add sequential edges in the initial undirected graph in ascending order of node number, connecting all nodes to obtain the final undirected graph.

[0057] In this optional embodiment, the hierarchical tree conversion module, through the collaborative work of the encoding submodule, the sibling edge addition submodule, and the sequential edge addition submodule, can effectively convert the code hierarchy tree into an undirected graph containing parent-child edges, sibling edges, and sequential edges. The encoding submodule provides a unified and meaningful identifier for nodes, the sibling edge addition submodule enhances the correlation between lines of code within the same level, and the sequential edge addition submodule ensures a complete representation of the linear execution flow of the code. This multi-dimensional and refined graph construction method allows the generated undirected graph to more comprehensively and accurately reflect the structure, logic, and sequential relationships of the source code, thus providing high-quality input for subsequent community detection algorithms. This significantly improves the accuracy and rationality of code semantic segmentation, solves the problem that traditional methods may lose key semantic information during graph construction, and makes the final code semantic segmentation result more consistent with the developer's understanding.

[0058] In an optional embodiment of this application, the community optimization module 40 includes: The first optimization submodule is used to disconnect the connection between two nodes with non-contiguous node numbers in a community if the node numbers within the community are not consecutive, thereby obtaining multiple sub-communities with consecutive node numbers.

[0059] The second optimization submodule is used to divide the community into multiple sub-communities if the community's occupied space is greater than the first preset occupied space, and the occupied space of each of the multiple sub-communities is less than the preset occupied space.

[0060] The Greedy Merge submodule is used to employ a greedy strategy to merge subcommunities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent.

[0061] In this optional embodiment, the first optimization submodule ensures that each code block is physically contiguous within the source code, providing a fundamental integrity for subsequent semantic analysis. Next, the second optimization submodule effectively controls the granularity of code blocks by splitting excessively large communities, preventing individual code blocks from containing too much irrelevant information and thus improving the focus of semantic analysis. Finally, the greedy merging submodule compensates for potential over-segmentation by merging excessively small adjacent communities, making the final code blocks more semantically complete and usable. This systematic optimization process allows the final code semantic segmentation results to better reflect the true semantic structure of the source code, improving the cohesion and independence of code blocks and providing high-quality input for advanced code analysis tasks such as code understanding, refactoring, and defect detection.

[0062] In an optional embodiment of this application, the second optimization submodule includes: If a community occupies more space than the first preset space, it is recorded as a community to be split. The last non-leaf node in the community to be split is used as the dividing point to split it into two sub-communities.

[0063] If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. The splitting continues with the last non-leaf node in the sub-community to be split as the dividing point until the space occupied by the resulting sub-communities is less than the preset space.

[0064] In this optional embodiment, when segmenting communities that occupy excessive space, the division is no longer done simply by randomness or fixed size. Instead, it is based on the structural information of the code hierarchy tree, using the last non-leaf node as the dividing point. This code logic structure-based splitting method effectively avoids truncating a complete code logic block in the middle, thus preserving the semantic integrity of the segmented sub-communities to the greatest extent. Furthermore, by continuously and recursively splitting until the space occupied by all sub-communities meets the preset requirements, the final code semantic segmentation result is ensured to meet size constraints while also possessing higher readability and maintainability. This makes subsequent code analysis, understanding, and refactoring more efficient and accurate, significantly enhancing the practical value of code semantic segmentation.

[0065] In summary, compared with existing technologies, it has the following beneficial effects: 1. Enhanced robustness: By reducing the reliance on complete and correct AST parsing, it can handle incomplete, fragmented, or non-standard code, which is particularly advantageous in practical scenarios such as code search and code completion, and its applicability far exceeds that of traditional AST block methods.

[0066] 2. Higher semantic coherence: Through community discovery algorithms, deep logical connections that are difficult to detect based on pure syntactic structure can be identified. The generated blocks have tighter semantic connections, providing more complete and context-rich input for subsequent retrieval and generation models.

[0067] 3. Flexibility and Adaptability: The community detection algorithm can adaptively generate blocks based on the inherent structure of the code, avoiding the mechanical nature that may exist in AST block division methods. By adjusting the weight definition of the edges in the graph and the parameters of the community detection algorithm, the granularity and preferences of the blocks can be flexibly controlled.

[0068] Specific implementation scenario example. Taking a piece of Python code as an example. This method first constructs a hierarchical tree based on indentation, with the root node being the module, and its child nodes being global variables, function definitions, etc. A function definition node contains child nodes such as parameters and statements within the function body. Subsequently, the community detection algorithm may identify that two syntactically independent functions that work together to complete a specific task (such as data validation) are grouped into the same community due to their proximity in the hierarchical tree and close logical connection, thus forming a more semantically complete block. In addition, for a piece of Python code with irregular syntax that cannot be parsed by a standard AST module, the AST-based method fails. However, the method in this application can still construct an effective hierarchical tree based on its indentation structure and generate meaningful code blocks that maintain internal logical coherence through the community detection algorithm, demonstrating the robustness of the method in this application.

[0069] Practical verification: In comparative experiments on Python codebases, compared with the cAST method, the proposed method significantly improves the Normalized Decay Cumulative Gain (NDCG) in code retrieval tasks, increasing by approximately 2.31 percentage points on NDCG@5 and 2.21 percentage points on NDCG@10 (NDCG@k is the NDCG value truncated to the first k results, where k=5 corresponds to NDCG@5 and k=10 corresponds to NDCG@10, indicating that only the ranking quality of the first 5 and 10 results in the recommendation / retrieval list is evaluated). This advantage is particularly pronounced when handling modules containing complex interactive logic. Furthermore, on test sets of code with intentionally introduced syntax errors, the proposed method achieves a significantly higher block segmentation success rate than the cAST method.

[0070] An electronic device provided in this application includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the code semantic block method based on hierarchical tree and graph community as described above when executing the computer program.

[0071] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the code semantic segmentation method based on hierarchical trees and graph communities as described above.

[0072] In this embodiment, the beneficial effects of the electronic device and the computer-readable storage medium are similar to those of the code semantic segmentation method based on hierarchical trees and graph communities described above, and will not be repeated here.

[0073] The present invention describes electronic devices that can serve as servers or clients of this application, which are examples of hardware devices that can be applied to various aspects of this application. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistant devices, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0074] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0075] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the separately described modules may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application according to actual needs. Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A code semantic segmentation method based on hierarchical trees and graph communities, characterized in that, include: Construct a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree. Lines of code with the same indentation level belong to the same indentation level. Each line of code is a node in a code hierarchy tree. Convert the code hierarchy tree into an undirected graph; A community detection algorithm is used to identify undirected graphs and segment them into multiple communities. Optimize multiple communities to obtain the code semantic block results.

2. The code semantic segmentation method based on hierarchical trees and graph communities as described in claim 1, characterized in that, The process of converting the code hierarchy tree into an undirected graph includes: In the code hierarchy tree, nodes are encoded, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code. Remove the root node, keep the parent-child edges, and add sibling edges between nodes in the same level according to their node numbers from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the connection between two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the connection between nodes in the same level in the code hierarchy tree. In the initial undirected graph, sequential edges are added in ascending order of node numbers to connect all nodes and obtain the final undirected graph.

3. The code semantic segmentation method based on hierarchical trees and graph communities as described in claim 1, characterized in that, The optimization process performed on multiple communities to obtain the code semantic segmentation results include: If the node numbers within a community are not consecutive, then the connection between two nodes with non-consecutive node numbers in the community will be broken, resulting in multiple sub-communities with consecutive node numbers. If the space occupied by the club is greater than the first preset space, the club will be divided into multiple sub-clubs, and the space occupied by each sub-club will be less than the preset space. A greedy strategy is adopted to greedily merge sub-communities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent, resulting in multiple final communities. The source code is then divided according to the node numbers within the final community to obtain the code semantic block result.

4. The code semantic segmentation method based on hierarchical trees and graph communities as described in claim 3, characterized in that, If the community's occupied space is greater than the first preset occupied space, then the community will be divided into multiple sub-communities, and the occupied space of each of the multiple sub-communities will be less than the preset occupied space, including: If the space occupied by a community is greater than the first preset space, it is recorded as a community to be split. The community to be split is split with the last non-leaf node as the dividing point to obtain two sub-communities. If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. The splitting continues with the last non-leaf node in the sub-community to be split as the dividing point until the space occupied by the resulting sub-communities is less than the preset space.

5. A code semantic block system based on hierarchical trees and graph communities, characterized in that, include: The hierarchical tree building module is used to build a code hierarchy tree based on the indentation level of each line of code in the source code. Each indentation level corresponds to a level in the code hierarchy tree, lines of code with the same indentation amount belong to the same indentation level, and each line of code is a node in a code hierarchy tree. The hierarchy tree conversion module is used to convert a code hierarchy tree into an undirected graph; The undirected graph segmentation module is used to identify undirected graphs using a community detection algorithm and segment the undirected graph into multiple communities. The community optimization module is used to optimize multiple communities and obtain code semantic block results.

6. The code semantic segmentation system based on hierarchical trees and graph communities as described in claim 5, characterized in that, The hierarchical tree conversion module includes: The encoding submodule is used to encode nodes in the code hierarchy tree, where the root node is a virtual node with a root node number of zero, and the node number is the line number of the code. The sibling edge is added as a submodule to remove the root node and retain the parent-child edge. Sibling edges are added between nodes in the same level according to their node numbers from smallest to largest to obtain the initial undirected graph. Here, the parent-child edge is the edge connecting two nodes in two adjacent levels in the code hierarchy tree; the sibling edge is the edge connecting nodes in the same level in the code hierarchy tree. The Sequential Edge Addition submodule is used to add sequential edges in the initial undirected graph in ascending order of node number, connecting all nodes to obtain the final undirected graph.

7. The code semantic segmentation system based on hierarchical trees and graph communities as described in claim 5, characterized in that, The community optimization module includes: The first optimization submodule is used to disconnect the connection between two nodes with non-continuous node numbers in a community if the node numbers in the community are not consecutive, so as to obtain multiple sub-communities with consecutive node numbers. The second optimization submodule is used to divide the community into multiple sub-communities if the community's occupied space is greater than the first preset occupied space, and the occupied space of each of the multiple sub-communities is less than the preset occupied space. The greedy merging submodule is used to employ a greedy strategy to merge sub-communities whose occupied space is less than or equal to the second preset occupied space and whose node numbers are adjacent, resulting in multiple final communities. The source code is then divided according to the node numbers within the final community to obtain the semantic code segmentation result.

8. The code semantic segmentation system based on hierarchical trees and graph communities as described in claim 7, characterized in that, The second optimization submodule includes: If the space occupied by a community is greater than the first preset space, it is recorded as a community to be split. The community to be split is split with the last non-leaf node as the dividing point to obtain two sub-communities. If the space occupied by either of the two sub-communities is greater than the first preset space, it is recorded as the sub-community to be split. The splitting continues with the last non-leaf node in the sub-community to be split as the dividing point until the space occupied by the resulting sub-communities is less than the preset space.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the code semantic segmentation method based on hierarchical trees and graph communities as described in any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the code semantic segmentation method based on hierarchical trees and graph communities as described in any one of claims 1 to 4.

Images