Code block segmentation method and device, program product, electronic equipment and storage medium

Abstract

The invention discloses a code block segmentation method and device, a program product, electronic equipment and a storage medium, and relates to the technical field of code segmentation. Aiming at the problem that semantic integrity and information density are difficult to consider in a traditional code segmentation method, the invention provides a code block segmentation method which is based on an abstract syntax tree and realizes hierarchical segmentation of codes through a father-son thumbnail partitioning strategy. According to the core idea, on the premise that code semantic units are kept complete, code blocks are subjected to recursive subdivision according to needs, and the code blocks are organized according to the set membership. When a certain code block is too large or contains a complex internal structure, the code block is split into a plurality of sub-blocks. The original parent block does not disappear, but is reserved in the hierarchical structure as an abstract parent block, structural information is reserved, and actual implementation details are borne by child node blocks. The AST provides structural guidance, father-child thumbnail partitioning ensures dynamic adjustment of partitioning granularity, and finally output data contain code content and have a hierarchical relationship.

Description

Technical Field

[0001] This invention relates to the field of code segmentation technology, and in particular to a code block segmentation method, apparatus, program product, electronic device, and storage medium. Background Technology

[0002] In the management and utilization of large-scale codebases, a key issue is how to divide the source code into small units suitable for retrieval and understanding. This is especially true in code retrieval-enhanced generation (Code-RAG) systems, where the segmentation of code snippets directly impacts the accuracy and efficiency of retrieval. By preprocessing the codebase into searchable fragments, retrieval-enhanced generation systems can quickly locate relevant context during code generation or question answering, improving the correctness and reliability of the generated results.

[0003] Current mainstream code segmentation methods include fixed-length segmentation, brace-based segmentation, and abstract syntax tree (AST)-based segmentation. However, these methods either ignore code structure or lack flexible granular control in code block partitioning, making it difficult to balance semantic integrity and information density. This directly leads to retrieval systems potentially retrieving irrelevant or incomplete code fragments, affecting downstream generation results.

[0004] Therefore, those skilled in the art urgently need a code block segmentation method to solve the problem that traditional code segmentation methods either ignore code structure or lack flexible granular control, making it difficult to balance semantic integrity and information density. Summary of the Invention

[0005] The purpose of this invention is to provide a code block segmentation method, apparatus, program product, electronic device and storage medium to solve the problem that traditional code segmentation methods are difficult to balance semantic integrity and information density.

[0006] To address the aforementioned technical problems, this invention provides a code block segmentation method, comprising: parsing the source code to construct an abstract syntax tree; determining the information density of each node in the abstract syntax tree; if the information density satisfies a first boundary condition, extracting the code corresponding to the node as a standard block; if the information density does not satisfy the first boundary condition, marking the node as a parent block node; extracting the structural information of the code corresponding to the parent block node, and creating a summary block containing the structural information as a parent block; determining the child nodes corresponding to the parent block node according to the node hierarchy and parent-child relationships between nodes in the abstract syntax tree, and extracting the code corresponding to the child nodes as child blocks; and segmenting the code according to the parent-child relationships between nodes in the abstract syntax tree. A standard block, the parent block, and the child blocks are linked to obtain a code block tree; wherein, the child blocks are attached to the corresponding parent blocks; the standard block, the parent block, and the child blocks all belong to code blocks; additional information of each code block in the code block tree is obtained, and the additional information is written as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation; wherein, the additional information includes: a unique identifier for the code block; for the parent block, the corresponding additional information also includes a list of child blocks, the list of child blocks including the unique identifier of each child block corresponding to the parent block; for the child block, the corresponding additional information also includes: a parent block identifier, the parent block identifier being the unique identifier of the code block corresponding to the parent block.

[0007] In an optional embodiment, after extracting the code corresponding to the child node as a child block, the method further includes: determining the information density of the child block; if the information density of the child block still does not meet the first boundary condition, then marking the node corresponding to the child block as a parent node, returning to the step of extracting the structural information corresponding to the parent block node, and creating a summary block containing the structural information as the parent block.

[0008] In one optional embodiment, determining the information density of each node in the abstract syntax tree includes: traversing each node in the abstract syntax tree from the root node downwards in a depth-first manner, and determining the information density of each node in turn.

[0009] In one alternative embodiment, determining the information density of the node includes: determining the number of lexical units contained in the code corresponding to the node.

[0010] In an optional embodiment, before determining the information density corresponding to each node in the abstract syntax tree, the method further includes: determining whether the node type of the node belongs to a predefined code block type; if yes, proceeding to the step of determining the information density of the syntax block corresponding to each node in the abstract syntax tree; if no, skipping this node, and performing the step of determining whether the node type of the node belongs to a predefined code block type on the child nodes of this node; wherein, the code block type includes: program structure unit, code logic unit, documentation and comment unit, and special language structure unit; the program structure unit includes: function definition, class / interface definition, module / file level definition; the code logic unit includes: main logic paragraph, anonymous code block, anonymous function expression; the documentation and comment unit includes: docstring or block comment attached to a function or class; the special language structure unit includes: specific language structure corresponding to a specific programming language, interface definition, enumeration definition.

[0011] In an optional embodiment, the method further includes: if the information density of any two adjacent code blocks satisfies the second boundary condition and belongs to the same code block type, then the two code blocks are merged; wherein the adjacent relationship includes: sibling adjacent relationship and / or physical adjacent relationship; the sibling adjacent relationship is: the two code blocks are sibling nodes in the code block tree; the physical adjacent relationship is: the two code blocks are consecutive in the source code.

[0012] In an optional embodiment, when the node is a child node, the code logic unit further includes: compound statement and code block nodes.

[0013] In an optional embodiment, the additional information further includes: the code block type.

[0014] In one optional embodiment, the step of parsing the source code to construct an abstract syntax tree includes: calling the corresponding abstract syntax tree parser to parse the source code according to the programming language corresponding to the file extension of the source code file, or the conventional programming language.

[0015] In one optional embodiment, the step of calling the corresponding abstract syntax tree parser to perform syntax parsing on the source code includes: calling the corresponding block type mapping table and performing syntax parsing on the source code through a unified abstract syntax tree parser; wherein, the abstract syntax tree parser uses a pre-abstracted generic block type; the block type mapping table stores the mapping relationship between the block type of the corresponding programming language and the generic block type.

[0016] In an optional embodiment, the method further includes: adding, deleting, or modifying the corresponding block type mapping table when adding, deleting, or modifying language support for the abstract syntax tree parser.

[0017] In an optional embodiment, the method further includes: dividing code segments in the source code that do not support syntax parsing by the abstract syntax tree parser into code blocks based on lines or symbols.

[0018] In one optional embodiment, the naming rule for the unique identifier of the code block includes: determining the corresponding unique identifier of the code block based on the position of the code block in the code block tree.

[0019] In an optional embodiment, the method further includes: establishing a corresponding index directory based on the position of the code block in the code block tree for the retrieval of the code block.

[0020] In an optional embodiment, the additional information further includes: summary information; wherein the summary information includes one or more of the following: the first line comment of the function in the code block, docstring, class inheritance relationship, and overall description information of the code fragment.

[0021] In an optional embodiment, the additional information further includes semantic tags; wherein the semantic tags include one or more of function names, parameter list summaries, and return types; the semantic tags are used as keywords for matching and retrieval in the code block retrieval.

[0022] In an optional embodiment, the additional information further includes: the start and end positions of the code block in the source code, and the file path of the source code to which it belongs; the start position, the end position, and the file path are used as attribute information in the retrieval of the code block for targeted retrieval.

[0023] In one optional embodiment, the source code includes: code stored in multiple source files; the code block graph is determined in parallel among different source files; and after the code block graph corresponding to each source file is determined, each code block graph is stored in a code library, and a correspondence is established between the code block graph and the source file, as well as with the version control system.

[0024] In an optional embodiment, the method further includes: when a change is detected in the source file, determining the changed source file for syntax parsing to construct a new abstract syntax tree; performing a difference comparison between the new and old abstract syntax trees to locate the changed nodes; re-determining the corresponding code block subtree for the changed nodes, and updating the code block tree corresponding to the changed source file based on the code block subtree.

[0025] In an optional embodiment, the method further includes: performing a consistency check on the code block tree; if the consistency check fails, recording the error and issuing an alarm; wherein the consistency check includes: traversing all code blocks in the code block tree and concatenating the code blocks to verify whether the concatenated code is consistent with the source code; checking whether the list of child blocks of the parent block is consistent with the unique identifier of the code block of its corresponding child block; and checking whether the unique identifier of the code block exists in the list of child blocks of the parent block and the unique identifier of the code block in the parent block identifier of the child block.

[0026] In an optional embodiment, after obtaining the code block map, the method further includes: storing the code block map in a vector retrieval database or a full-text search engine; wherein each code block in the code block map is treated as a file entry; the vector retrieval database and the full-text search engine provide query services to the outside world through a retrieval interface; when a code query request is received, the retrieval interface is called to query the corresponding code block.

[0027] In one optional embodiment, the code query request includes one or more of a standard block query request, a parent block query request, and a child block query request. For the standard block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the standard block query request, and returning the content text of the target code block as the query result. For the parent block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the parent block query request, and returning the content text of the parent or ancestor block corresponding to the target code block as the query result. The ancestor block is the code block in the code block graph whose parent-child relationship is above the parent block of the target code block. For the child block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the child block query request, and returning the content text of the child or descendant block corresponding to the target code block as the query result. The descendant block is the code block in the code block graph whose parent-child relationship is below the child block of the target code block.

[0028] In an optional embodiment, if multiple code blocks are retrieved according to the code query request, the method further includes: sorting and assembling the retrieved code blocks according to their position order in the source code to obtain a code segment set, and generating corresponding context prompt information; and returning the code segment set and the context prompt information as query results.

[0029] To address the aforementioned technical problems, this invention also provides a code block segmentation device, comprising: a syntax parsing module for parsing source code to construct an abstract syntax tree; a density determination module for determining the information density of each node in the abstract syntax tree; a standard extraction module for extracting the code corresponding to the node as a standard block if the information density satisfies a first boundary condition; a parent block extraction module for marking the node as a parent block node if the information density does not satisfy the first boundary condition; extracting the structural information of the code corresponding to the parent block node and creating a summary block containing the structural information as a parent block; a child block extraction module for determining the child node corresponding to the parent block node according to the node hierarchy and parent-child relationship between nodes in the abstract syntax tree, and extracting the code corresponding to the child node as a child block; and a block tree generation module for... Based on the parent-child relationships between nodes in the abstract syntax tree, the standard block, the parent block, and the child block are linked to obtain a code block tree; wherein, the child block is attached to the corresponding parent block; the standard block, the parent block, and the child block all belong to code blocks; the graph construction module is used to obtain additional information of each code block in the code block tree, and write the additional information as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation; wherein, the additional information includes: a unique identifier for the code block; for the parent block, the corresponding additional information also includes a list of child blocks, the list of child blocks including the unique identifier of each child block corresponding to the parent block; for the child block, the corresponding additional information also includes: a parent block identifier, the parent block identifier being the unique identifier of the code block corresponding to the parent block.

[0030] To address the aforementioned technical problems, the present invention also provides a computer program product, comprising a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the steps of the code block segmentation method described above.

[0031] To address the aforementioned technical problems, the present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the code block segmentation method described above when executing the computer program.

[0032] To address the aforementioned technical problems, the present invention also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps of the code block segmentation method described above.

[0033] This invention provides a code block segmentation method. First, it uses an Abstract Syntax Tree (AST) to perform preliminary semantic segmentation of the source code, avoiding damage to the code structure and semantics. Then, it determines the information density of each code block (i.e., each node in the AST) obtained after the preliminary segmentation. If the information density meets a first boundary condition, it indicates that the information density of the code block is not too large and can be directly extracted as a standard code block (i.e., a standard block). If the information density does not meet the first boundary condition, it indicates that the information density of the code block is too large, potentially exceeding the context window of the downstream model or reducing the retrieval accuracy of the code block. In this case, this method marks it as a parent block (the node is marked as a parent node). The parent block only retains the structural information of the code, but not the specific code. The structural information, as a code summary, can still reflect the overview and context of the parent block's code. Not retaining the specific code avoids the problems caused by an excessively large information density in a code block. Furthermore, the parent block can be further split through its lower-level child nodes in the AST, that is, a large parent block can be split into multiple smaller child blocks. Each sub-block can contain specific code, and when pieced together, they form the complete code content of the parent block, reflecting the code details of the parent block and ensuring no code content is lost. Furthermore, after splitting the code into blocks, this method adds a unique identifier (ID) to each block. On one hand, the code block ID facilitates subsequent code block retrieval. On the other hand, each corresponding sub-block and parent block includes the other's code block ID, allowing the parent-child relationship to be directly represented in the code block graph. In subsequent code block retrieval, if the query target is a parent block with high information density, all sub-blocks corresponding to that parent block can be returned as query results based on this parent-child relationship, ensuring no loss of parent block code details. Conversely, if the query target is a split sub-block, the corresponding parent block can also be returned based on this parent-child relationship to obtain the overall concept of the class or function corresponding to that sub-block. In other words, this method provides a code block splitting method based on AST but with adaptive scaling of splitting granularity, thus combining the advantages of AST splitting while overcoming its inflexible splitting granularity, achieving a superior code block splitting scheme.

[0034] The code block splitting device, computer program product, electronic device and non-volatile storage medium provided by the present invention correspond to the above methods and have the same effect. Attached Figure Description

[0035] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1A flowchart of a code block segmentation method provided in an embodiment of the present invention.

[0037] Figure 2 This is a hardware implementation architecture diagram of a code block segmentation method provided in an embodiment of the present invention.

[0038] Figure 3 This is a structural diagram of a code block splitting device provided in an embodiment of the present invention.

[0039] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0041] The core of this invention is to provide a code block segmentation method, apparatus, program product, electronic device, and storage medium.

[0042] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] In related technologies, the mainstream code block splitting schemes mainly include fixed-length splitting, brace-based splitting, and abstract syntax tree (AST)-based splitting.

[0044] 1. Fixed-length segmentation: This method divides the code into blocks based on a fixed number of characters or lines. It is simple to implement and widely used in general text processing. However, for structured code, this coarse-grained segmentation often disrupts the code's semantic structure. For example, fixed-size segmentation might truncate a function in the middle, causing the first and second halves of a function to be assigned to different segments. Incomplete code segments are difficult to retrieve correctly and may cause downstream generative models to lose important contextual information, resulting in fragmented semantic units and reduced retrieval and generation quality.

[0045] 2. Bracket-based segmentation: In programming languages ​​such as C, C++, and Java, which use curly braces to define scope, a common practice is to divide code blocks by curly braces, for example, treating the curly brace range of each function or class as a single segment. This method preserves the structural integrity of the code to some extent, avoiding the splitting of explicitly defined code blocks. However, its limitation is that it cannot be directly applied to programming languages ​​without explicit curly braces (such as Python); also, if the code within a curly brace range is too large (for example, a function containing hundreds of lines of code), treating the entire function as a single segment may still be too large, exceeding the context window of the downstream model or reducing retrieval accuracy. Furthermore, segmenting only by curly braces cannot further refine the code's logical structure; for example, logical paragraphs within a function are still mixed in one block, which is not conducive to fine-grained retrieval.

[0046] 3. AST-based segmentation: This method uses an abstract syntax tree to parse the code structure, dividing it into units based on syntax nodes. For example, each function definition or class definition can be treated as a code fragment, or each module in Python can be treated as an independent unit. AST segmentation methods can identify the hierarchical structure of the code, preserving complete syntax units, unlike fixed-length segmentation which arbitrarily breaks functions or classes. However, existing AST-based segmentation is usually performed at a predefined granularity (such as function-level or class-level), which has limitations in granularity selection: if the granularity is too coarse, large functions are still difficult to process; if the granularity is too fine (e.g., each statement is treated as a fragment), too many fragments will be generated, affecting retrieval efficiency. In addition, different languages ​​have different AST structures, often requiring customized parsing rules during implementation, lacking a universal cross-language solution.

[0047] As can be seen from the above, several mainstream code block segmentation schemes each have their own advantages and disadvantages. Some ignore code structure, while others lack flexible granular control, and there is a lack of a code block segmentation method that can balance semantic integrity and information density. To solve this problem, this invention provides a code block segmentation method, such as... Figure 1 As shown, it includes: S101: performing syntax parsing on the source code to construct an abstract syntax tree.

[0048] S102: Determine the information density of each node in the abstract syntax tree.

[0049] S103: If the information density satisfies the first boundary condition, then extract the code corresponding to the node as a standard block.

[0050] S104: If the information density does not meet the first boundary condition, mark the node as a parent block node; extract the structural information of the code corresponding to the parent block node, and create a summary block containing the structural information as the parent block.

[0051] S105: Based on the node hierarchy of the abstract syntax tree and the parent-child relationship between nodes, determine the child nodes corresponding to the parent block node, and extract the code corresponding to the child node as the child block.

[0052] S106: Based on the parent-child relationships between nodes in the abstract syntax tree, link the standard block, parent block, and child block to obtain the code block tree.

[0053] In this context, child blocks are attached to their corresponding parent blocks; standard blocks, parent blocks, and child blocks all belong to code blocks.

[0054] S107: Obtain additional information for each code block in the code block tree, and write the additional information as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation.

[0055] The additional information includes: a unique identifier (ID) for the code block; for the parent block, the corresponding additional information also includes a list of child blocks, which includes the unique identifier of each child block corresponding to the parent block; for the child block, the corresponding additional information also includes: the parent block identifier, which is the unique identifier of the corresponding parent block.

[0056] Regarding step S101 above, the specific implementation of semantic parsing and decomposition of the code using AST can be found in the description of related technologies above, and will not be repeated here. The purpose of step S101 is to perform preliminary code block decomposition using AST to obtain preliminary code blocks (one preliminary code block corresponds to one node in the AST) that retain the hierarchical structure and complete syntactic units of the code. Specifically, when determining the boundary position during code block decomposition, this step strictly follows the syntactic structure of the AST and will not separate content that syntactically belongs to the same node. In other words, block boundaries can only appear at AST node boundaries and will not appear at arbitrary code locations. This ensures that each code block is semantically consistent. For example, when a function is decomposed, it can only be decomposed at the statement boundaries within the function body, and a compound statement will not be truncated. Through AST node boundary constraints, this method avoids the semantic damage caused by traditional fixed partitioning. The boundaries between blocks are naturally consistent with the structured boundaries of the source code, ensuring structural integrity.

[0057] Then, the core of this method, steps S102 to S105, further splits the code blocks based on the parent-child abbreviation block strategy.

[0058] Specifically, step S102 first determines whether the information density of each preliminary code block after initial splitting meets the requirements. The requirement for information density is manifested as meeting a first boundary condition, which should be specifically determined based on the information density requirements of the code block splitting; this embodiment does not impose any restrictions on this. For example, assuming the number of lines or characters of code is used as the unit of measurement for code block information density, the corresponding first boundary condition can be a specific threshold. If the information density is less than or equal to this threshold, it is considered to meet the first boundary condition; otherwise, it is considered not to meet the first boundary condition.

[0059] Furthermore, regarding whether the first boundary condition in step S102 is satisfied, there are two different branches: steps S103 and S104. For branch one, i.e., step S103, the information density of the code block initially split based on the AST meets the requirements and can be directly extracted as a code block output for subsequent processing. For ease of distinction, in this embodiment, the code block that meets the information density requirements after the initial AST split is called a standard (code) block. As for branch two, i.e., steps S104 and S105, it targets code blocks whose information density does not meet the requirements after the initial AST split. The information density of such code blocks is too high, and if the code is directly extracted, it may exceed the context window of the downstream model or reduce the retrieval accuracy of the code block. Therefore, steps S104 and S105 in this method further split them.

[0060] In step S104, these code blocks (nodes) are marked as parent blocks (parent block nodes). Only the structural information of the corresponding code block is retained in the parent block; the specific code content is not retained. That is, from the perspective of its content, this parent block is essentially a summary block. However, considering that the complete code of the parent block can also be fully presented by concatenating its child blocks, the parent block will also be considered a code block in the following description. It should be noted that this embodiment does not limit the specific content of the structural information, which may include, but is not limited to: class names / function names, parameters, documentation comments, and other information that can help understand the parent block code; it is essentially a code summary. The purpose of this step is to retain the summary of the parent block code while avoiding the problem of excessive information density in the parent block, that is, to retain the overall concept of the parent block as a large code block and not omit any semantic units.

[0061] Next, regarding step S105, to avoid losing specific code content, this method splits the parent block into multiple child blocks in step S105, where the child blocks retain the specific code, thus reflecting the code details of the parent block. Specifically, when performing semantic parsing of source code based on AST, AST can determine the parent-child relationship between code blocks based on the semantic relationships between the code. Therefore, after determining the parent block, the code blocks of the child nodes corresponding to the parent block nodes can be used as the child blocks corresponding to the parent block based on the parent-child relationships in the AST.

[0062] Based on this, this method further splits parent blocks with excessive information density into multiple child blocks with lower information density. The parent block retains only a code summary, not the actual code content. The child blocks retain the actual code content, and concatenating these child blocks yields the complete parent block code without losing any code content. In other words, this method provides a parent-child abbreviated block splitting strategy that further splits excessively dense code blocks (parent blocks) into multiple child blocks at the code detail level, while still preserving the overall concept (the parent block retains structural information), thus balancing the preservation of code semantics and overall architecture. The clarity of this split code block structure far surpasses that of a fixed-size fragment list, facilitating manual browsing and understanding, and providing contextual clues for retrieval.

[0063] Furthermore, to clearly characterize the relationship between parent and child blocks and achieve the structural advantages of the code block splitting mentioned above during retrieval, this method also proposes steps S106 and S107. Step S106's core lies in injecting additional metadata into the further split code blocks (standard block, parent block, and child block). This additional information (i.e., the injected metadata) includes at least the code block ID corresponding to the code block, used to uniquely identify the code block. For parent and child blocks, the additional information also includes the other's code block ID, also used to characterize the parent-child relationship between the code blocks. It should be noted that generally, one child block corresponds to only one parent block, but one parent block can correspond to multiple child blocks. Therefore, the additional information of the parent block contains a list of child blocks (i.e., a set of corresponding child block IDs), and the additional information of the child blocks contains the parent block ID. Based on this setup, the corresponding code block can be retrieved efficiently and conveniently using its code block ID during code block retrieval. Simultaneously, when this code block is a parent block, the corresponding child block ID can be found through the child block list in its additional information. If you need to query the code details of a parent block, you can find the corresponding child blocks by their IDs, extract the child block code, and concatenate it to obtain the complete parent block code. Similarly, when the queried code block is a child block, if you want to know the overall concept of this child block's class / function, you can also find the structure information stored in the corresponding parent block by its ID in the additional information, and return it as a code summary.

[0064] Furthermore, step S107 provides a persistent storage scheme for each partitioned code block. After additional information is written, each code block in the aforementioned code block tree can more directly and comprehensively represent the parent-child relationship between code blocks, which can be regarded as a code block graph. In this graph, nodes represent specific code blocks, and edges represent the parent-child relationship between code blocks. The code block graph displays the hierarchical relationship between the code blocks obtained from source code switching and can be used for subsequent retrieval.

[0065] In summary, this application provides a code block segmentation method based on AST but with adaptively scaling granularity. It combines the advantages of AST segmentation while overcoming the inflexible granularity of traditional AST segmentation schemes, implementing a parent-child abbreviated block segmentation scheme. The higher-level parent block retains only structural information, providing the overall concept and context of the class / function. However, the parent block does not retain the specific code content; the lower-level child blocks provide the specific code details. In other words, this method achieves hierarchical semantic representation, making the segmented code block structure clearer and facilitating subsequent retrieval or viewing with flexible granularity scaling.

[0066] On the other hand, as can be seen from the above embodiments, this method solves the problem of excessive information density in parent blocks whose information density does not meet the first boundary condition by splitting them into sub-blocks. However, in some special scenarios, the split sub-blocks may still not meet the first boundary condition. In this case, this embodiment also provides a solution accordingly. After step S105: extracting the code corresponding to the child node as a sub-block, the above method further includes: S108: determining the information density of the sub-block; if the information density of the sub-block still does not meet the first boundary condition, then the node corresponding to the sub-block is marked as the parent node, and the process returns to step S104.

[0067] In this embodiment, a sub-block can be further divided until all the resulting code blocks satisfy the first boundary condition. It's important to note that the standard block, parent block, and child block mentioned above are only one type of identity for a code block, and a code block may have multiple identities. For example, in this embodiment, all child blocks should also have the identity of a standard block because they satisfy the first boundary condition. A code block may have both the identity of a parent block and the identity of a child block. For instance, a branch in the code block tree, based on parent-child relationships, contains the following three code blocks from top to bottom: code block A—code block B—code block C. In this branch, code block B is both a child block of code block A and a parent block of code block C. Furthermore, code block A can be called the ancestor block of code block C, and code block C can be called a descendant block of code block A.

[0068] It should also be noted that when a code block simultaneously has the identity of a parent block and a child block, the division of this code block is based on the parent block; that is, only structural information is retained, not specific code information. In this case, the code details represented by this code block can be provided by its set of child blocks. For example, taking the code blocks A, B, and C mentioned above, the code details in code block A that originally needed to be provided by child block B can be further provided by child block C of code block B (this example temporarily ignores other child blocks of code blocks A and B). Similarly, the principle of the three-level parent-child relationship in the above example can also be applied to more levels of parent-child relationships, which will not be elaborated in this embodiment.

[0069] Based on the implementation scheme provided in this embodiment, more levels of code block division can be achieved. This ensures that each code block meets the information density requirements while also preventing the loss of any semantic units in the source code.

[0070] On the other hand, after the above embodiments have described in detail the core part of this method, namely the parent-child abbreviation block strategy, this embodiment will further explain the remaining parts of this method. First, regarding the code block operations involved in steps S101 to S105 of the above method, the specific code block operations can be implemented through a combination of a series of pipeline processing steps and recursive algorithms.

[0071] For example, this embodiment provides an optional implementation scheme for the specific block implementation of step S101. Step S101 specifically involves: traversing each node in the abstract syntax tree from the root node downwards in a depth-first manner, and determining the information density of each node in turn.

[0072] Furthermore, in this embodiment, a depth-first traversal begins from the root node of the AST. Before entering a child node, it can be determined whether the current node itself constitutes a block and whether its size requires further splitting (i.e., marked as a parent block split into child blocks). This strategy of splitting into blocks first and then delving deeper ensures that higher-level semantic units are prioritized into blocks, and boundary judgments are not interfered with by excessively fine nodes at lower levels. For example, a large class will first be identified as a block that needs to be split, and then its internal methods will be processed, without ignoring the boundary of the class level because there are more small nodes inside the methods. This mechanism of judging the parent node first and then processing the child nodes enables this method to include complete large units into a block as much as possible, and only split them when necessary.

[0073] This embodiment utilizes the aforementioned pipeline + recursion approach to provide an optional scheme for initial code block partitioning based on AST. The reason for using depth-first search instead of breadth-first search or other alternative traversal schemes is that depth-first search better supports top-down, recursive processing of nested structures and is more suitable for splitting parent-child code blocks. This embodiment employs a top-down traversal method starting from the root node to maximize the inclusion of large nodes into single code blocks, thereby reducing the number of code blocks and lowering the complexity of the partitioning structure. Furthermore, as demonstrated in the previous embodiment, if a code block is too large, it can be further partitioned based on a parent-child abbreviated block strategy. Therefore, this traversal method does not result in excessively high information density in a single code block. Additionally, based on this top-down traversal scheme, subsequent combinations of child blocks to form parent blocks also achieve a bottom-up hierarchical structure, making the entire code block partitioning structure clearer and more readable.

[0074] On the other hand, an important criterion used in the above embodiments for dividing code blocks is whether the information density of the code block meets the first boundary condition. Regarding information density, the above embodiments provide a general implementation scheme, which uses the number of lines or non-whitespace characters contained in the code block as a measure of information density. However, this scheme may lead to misjudgment due to differences in the format used in the source code. Furthermore, this measurement method is not very comparable when the source code uses different programming languages. To address this, this embodiment provides an optional implementation scheme. Step S102: Determining the information density of a node specifically includes: determining the number of tokens contained in the code corresponding to the node. Here, a token is the smallest syntactic unit in the code, such as a keyword, identifier, literal, operator, delimiter, etc. This can avoid misjudgment caused by formatting or blank lines, ensuring the comparability of information density across different languages ​​and styles. Further, based on this embodiment, an optional scheme for the first boundary condition is: the number of tokens is less than or equal to 512.

[0075] On the other hand, to achieve more general and accurate code segmentation, this embodiment also provides a further implementation scheme. Before step S102, the above method further includes: S109: determining whether the node type of the node belongs to a predefined code block type; if yes, proceed to step S102; if no, proceed to step S110.

[0076] S110: Skip this node and perform step S109 on the child nodes of this node.

[0077] The code block types include: program structure units, code logic units, documentation and comment units, and special language structure units.

[0078] Program structure units include: function definitions (including methods, constructors, etc.), class / interface definitions, and module / file-level definitions. These are usually self-contained logical units in the code, with independent meaning, and are suitable as retrieval granularities. For example, in Java, classes and methods are main blocks, in Python, functions and classes are also blocks, and in C, functions are block nodes.

[0079] The code logic unit includes: main logic paragraphs, anonymous code blocks, and anonymous function expressions (Lambda expressions). It should be noted that this type is optional. In some cases, logical segments within a function can also be considered sub-blocks, such as main logic paragraphs, anonymous code blocks, and Lambda expressions in a large function. If the target function code body is too long, it may need to be further split along semantic segments. In this case, statement block nodes in the AST (such as compound statements and code block nodes) can also be included in the block type, but they are usually only extracted as sub-blocks when the parent block needs to be split. That is, this embodiment further provides an optional implementation: when a node is a child node, the code logic unit also includes: compound statements and code block nodes. For example, in the AST representation, an if-else structure or a for loop and its internal statements can be classified as a sub-block, belonging to the function block it belongs to.

[0080] Documentation and comment units include: docstrings or block comments preceding functions or classes. Specifically, the docstrings or block comments preceding functions or classes are semantically closely related to the code unit. In this embodiment, they are considered part of the block content rather than an independent block. However, for the metadata of code snippets, whether or not there is documentation is recorded separately, and comments are included in the summary when necessary to aid retrieval and understanding. For example, in Python, the docstring of a function forms a code block together with the function body; the Javadoc comments preceding Java methods are also incorporated into the summary information of the method block.

[0081] Special language structural units include: language-specific structures corresponding to a particular programming language, interface definitions, and enumeration definitions. Specifically, this type refers to specific structures existing in certain languages, such as the `impl` block (a keyword used to define an implementation block) in Rust (a programming language) and attribute annotation areas in C#. If they have independent meaning, they can also be considered block nodes. Furthermore, interface definitions and enumeration definitions are also categorized as structural unit blocks.

[0082] As described above, in order to achieve general and accurate code segmentation, this embodiment first defines the range of block (node) types, that is, which node types in the AST will be considered as units that can form independent blocks. Defining block (node) types requires comprehensive consideration of the commonalities and unique structures of various languages, and follows the principles of semantic integrity and retrieval usefulness. The definition scheme provided in this embodiment well meets this requirement.

[0083] Furthermore, based on the above embodiments, it is clear that the type of each code block can effectively reflect whether the code blocks have the same semantics. Therefore, this embodiment also provides a further implementation scheme, the method further including: S111: If the information density of any two adjacent code blocks satisfies the second boundary condition and belongs to the same code block type, then the two code blocks are merged.

[0084] The adjacency relationship includes: sibling adjacency and / or physical adjacency; sibling adjacency means that the two code blocks are sibling nodes in the code block tree; physical adjacency means that the two code blocks are consecutive in the source code.

[0085] First, it should be noted that this embodiment does not limit the specific conditions of the second boundary condition. However, based on the above embodiments, it can be understood that this embodiment determines "too small" code blocks based on the second boundary condition and merges them to reduce the number of code blocks. The first boundary condition is used to determine "too large" code blocks. Therefore, to ensure that both boundary conditions can fully play their role, the first boundary condition should include the second boundary condition. Or, in a more easily understood way, the second boundary condition can be considered "smaller" than the first boundary condition. Furthermore, the merging of code blocks based on the second boundary condition should also avoid situations where re-splitting is required. That is, after merging two code blocks that satisfy the second boundary condition, the first boundary condition should also be satisfied; otherwise, it will trigger the re-splitting of code blocks, wasting resources and potentially causing the code block splitting to fall into an infinite loop. Based on this, it can be further concluded that twice the second boundary condition is "smaller" than the first boundary condition.

[0086] For example, taking the above embodiment where information density is measured by the number of tokens and the first boundary condition is that the number of tokens is less than or equal to 512, an optional second boundary condition can be that the number of tokens is less than or equal to 50. 2*50<512, which satisfies the requirement that the second boundary condition, which is twice as much as the first boundary condition.

[0087] It should also be noted that this step can utilize the sibling adjacency relationship of nodes in the code block tree. Therefore, step S111 provided in this embodiment can occur during the process of obtaining the code block tree in step S106, that is, the merging of adjacent and semantically similar small code blocks is completed simultaneously when the code block tree is generated. It can also occur after step S106, that is, after the code block tree is generated, the merging of adjacent and semantically similar small code blocks is performed. This embodiment does not limit this.

[0088] In this embodiment, multiple adjacent small blocks can be merged into a larger block to increase the information density of a single block. For example, a series of consecutive very short function definitions can be merged into a module-level code block.

[0089] On the other hand, after providing a specific definition of a code block type in the above embodiments, the approximate semantics of each code block can be conveniently and efficiently determined based on this code block type. This function can also be well applied in the subsequent retrieval process of code blocks. In addition, the above embodiments also illustrate that the additional information (metadata) in this invention is not limited to the code block ID, sub-block list, and parent block ID mentioned above. Therefore, based on this, this embodiment also provides an optional implementation scheme for additional information: the additional information also includes the code block type. That is, in this embodiment, the code block type is also written into the code block as the metadata of the code block. When a code block retrieval is required subsequently, the corresponding code block can be retrieved according to semantic requirements, providing a more diverse and flexible retrieval scheme for code block retrieval.

[0090] Furthermore, as explained in the above-mentioned technical sections, a significant problem with common code partitioning schemes based on Abstract Syntax Trees (ASTs) is that different languages ​​have different AST structures, often requiring customized parsing rules and lacking a universal cross-language solution. To address this issue, this embodiment provides a multi-language adaptation solution. Step S101 specifically involves: based on the programming language corresponding to the file extension of the source code file, or a conventional programming language, calling the corresponding Abstract Syntax Tree parser to perform syntax parsing on the source code.

[0091] However, providing a corresponding AST parser for each different programming language would make the entire implementation difficult and cumbersome. Therefore, this embodiment also provides a further implementation scheme. Step S101 of the above embodiment: calling the corresponding abstract syntax tree parser to perform syntax parsing on the source code, specifically includes: calling the corresponding block type mapping table and performing syntax parsing on the source code through a unified abstract syntax tree parser. The abstract syntax tree parser uses pre-abstracted general block types; the block type mapping table stores the mapping relationship between the block types of the corresponding programming language and the general block types.

[0092] Specifically, this embodiment defines block types within a common framework and then maps them to the AST node types for each language. For example, taking the program structure unit, one of the general code block types abstracted in the above embodiment, as an example, it includes: modules / files, type definitions (such as classes / interfaces / enumerations), function / method definitions, and code segments / logic blocks. In this case, a block type mapping table can be established for each programming language.

[0093] 1. Java / C++ / C#, etc.: Map the "class (class / struct)" of the AST node to the type definition block, the "function / method (function / method)" to the "function" block, and the namespace or package as the "module" block. If there are anonymous inner classes inside, they are also treated as blocks as needed.

[0094] 2. Python: Modules are treated as "module" blocks, ClassDefs as "class" blocks, and function definitions (FunctionDef) and asynchronous function definitions (AsyncFunctionDef) as "function" blocks. Because Python uses indentation and allows functions or classes to be defined within functions, the recursive algorithm used in the above embodiments of this invention is naturally suitable for handling these nested cases, and internal definitions will be treated as sub-blocks. Top-level flow control (such as the top-level `if name == 'main'`) is considered a logic block.

[0095] 3. JavaScript / TypeScript: Define top-level functions and classes as blocks. Considering JavaScript's functional style and module mechanism, this embodiment can also treat files as "module" blocks. Immediately Invoked Function Expressions (IIFEs) or Lambda expressions can also be included in logic blocks if needed.

[0096] 4. Other programming languages: For example, Go's `func` (a keyword used to claim a function), Rust's `fn` (a keyword used to claim a function) and `impl` (a keyword used to claim an implementation block) all follow similar rules. In short, by adjusting the mapping table, new language parsing can be quickly supported without changing the chunking algorithm process.

[0097] This embodiment can determine the corresponding block type mapping table based on the file extension or convention when loading source code files, mapping the special block types corresponding to each programming language to a pre-abstracted general block type. Thus, semantic parsing of source code from multiple different programming languages ​​can be achieved based on an AST parser implemented using a general block type. Furthermore, this embodiment shields programming language differences through the AST parsing layer, achieving consistent behavior across languages. Subsequent node traversal and block splitting do not need to consider programming language issues, greatly simplifying implementation and improving execution efficiency.

[0098] On the other hand, there are special language structural units in the code block type classification as described in the above embodiments. Some programming languages ​​have their own special structures or conventions. Therefore, the block segmentation strategy can also be adjusted accordingly for these programming languages ​​to better adapt to different programming languages ​​and achieve accurate and reliable code block segmentation.

[0099] 1. For scripting languages ​​(such as Python and JavaScript): Since these programming languages ​​allow execution statements to be written directly at the top level, the set of top-level statements can be regarded as a special "module code segment" block in order to preserve the context of the script execution flow.

[0100] 2. For statically typed languages ​​(such as C++ and Java): These programming languages ​​may contain preprocessing structures such as macros and templates. These preprocessing structures can be expanded as much as possible during the AST parsing stage to obtain a unified AST; if complete expansion is not possible, macros can be segmented into independent blocks according to their structure, or marked as unparseable blocks for special handling (see the subsequent examples on how to handle code segments that do not support AST parsing).

[0101] 3. For multi-paradigm languages ​​(such as Scala, which supports both object-oriented and functional programming): the block type mapping can simultaneously include nodes such as classes, object definitions, and functions. Furthermore, closures and lambda expressions in some programming languages ​​are typically expression nodes in the AST, so this method generally does not treat them as top-level blocks when splitting code blocks, unless their size is large enough to be considered as a sub-block of the function they belong to.

[0102] 4. Regarding comments and documentation: Different languages ​​use different methods to extract comments, so they should be handled accordingly. For example, Python's docstring can be directly obtained as expression (Expr) nodes through the AST, while Java's Javadoc needs to be extracted and appended to the AST node during the lexical stage. They can be processed separately, but a unified metadata field should be provided in the result.

[0103] Furthermore, this embodiment provides a further implementation scheme based on the above embodiments. The method further includes: S201: When adding, deleting, or modifying language support in the abstract syntax tree parser, performing addition, deletion, or modification operations on the corresponding block type mapping table. It is easy to see from the above that the multi-language support scheme provided by the above embodiments not only simplifies the implementation difficulty of the overall scheme when supporting multiple languages, but also makes it simpler and more efficient to modify the supported languages. This embodiment does not require modification of the AST parser; only corresponding modification operations are needed on the block type mapping table, i.e., adding, deleting, or modifying the mapping relationship with the general block type, which is easy to implement.

[0104] Based on the multi-language adaptation solution provided in the above embodiments, the goal of "one framework, universal for multiple languages" has been achieved. From the user's perspective, regardless of whether the codebase contains Java, Python, or other languages, it can be retrieved and managed through a unified hierarchical block-based result. This greatly reduces the difficulty of obtaining context in multi-language projects and improves the system's versatility and practical value.

[0105] On the other hand, as can be seen from the above embodiments, this method still uses AST as the initial code block partitioning scheme in order to leverage the advantage of AST in preserving the semantic integrity of the code. However, code block splitting based on AST may also have some special cases: 1. There are empty code segments or very small code segments (such as empty functions, empty classes, or blocks containing only a few simple statements); 2. There are code segments that do not support AST parsing (such as unparseable code or binary macro expansion parts). Among them, the above-mentioned special case 1 can be solved by the adjacent code block merging scheme provided in the aforementioned embodiments.

[0106] Regarding the special case 2 mentioned above, this embodiment also provides an alternative solution. The method further includes: S202: For code segments in the source code that do not support syntax parsing by an abstract syntax tree parser, divide them into code blocks based on lines or symbols. It should be noted that this embodiment provides a fallback strategy for code segments that do not support AST. That is, the boundaries can be approximately defined by lines or symbols to isolate unparseable parts into separate blocks as much as possible, so as not to affect the division of other code parts.

[0107] By combining the solutions provided in this embodiment and those in the above embodiments, this method can accurately identify each suitable boundary point when dividing code blocks: it will neither miss internal units that should be divided into blocks nor forcibly break up content that should not be divided. The resulting code blocks have clear and reasonable boundaries, and each block corresponds to a complete paragraph or structure in the source code, with a clear start and end range and contextual meaning.

[0108] On the other hand, the above embodiments illustrate that code block IDs can be used to uniquely identify each code block, and also to determine the parent-child relationship between code blocks. However, there are no strict restrictions on how code block IDs are generated, as long as they are uniquely matched. Based on this, this embodiment provides an optional code block ID naming rule: determining the unique identifier of the corresponding code block based on its position in the code block tree.

[0109] In other words, this embodiment utilizes the position of code blocks in the code block tree to provide a hierarchical encoding scheme. As can be seen from the above embodiment, code block types include module or file-level definitions, class or interface-level definitions, and function or method-level definitions. The sizes of these different code block types are not necessarily the same. For example, a possible code block, according to the above naming rules, might be hierarchically encoded as 1.2.3, representing the third method block of the second class in the first file. This ID can intuitively reflect the position of the code block in the code block tree, facilitating quick location of parent and child nodes.

[0110] Furthermore, this embodiment also provides a further implementation scheme based on the above embodiment. The above method further includes: S203: establishing a corresponding index directory according to the position of the code block in the code block tree for code block retrieval.

[0111] Specifically, this embodiment can employ a hierarchical index storage based on the location hierarchy of the code blocks, i.e., building an index directory level by file / module, class, and function. For example, in a database, one can first index by file ID, then by class ID under the file, and finally query the corresponding code block ID based on the lower-level function ID. This storage structure naturally corresponds to the code block tree, enabling efficient hierarchical queries. Furthermore, as a type of metadata, the code block ID can be queried through a unified data interface, providing the metadata to external systems via object models or JSON / XML (both data exchange formats). In addition, when implementing the index in practical applications, full-text indexes can be built for commonly used search fields (such as name, summary, docstring, etc.), supporting quick retrieval of relevant code block nodes by keywords or symbol names.

[0112] In addition to the examples given above, the metadata (additional information) of the code block can also include other information as needed. For example, this embodiment also provides an optional implementation: the additional information further includes summary information; wherein the summary information includes one or more of the following: the first line comment of the function in the code block, the docstring, the class inheritance relationship, and the overall description information of the code fragment.

[0113] The purpose of adding summary information to the supplementary information in this embodiment is to allow users to understand the general content of the code block without viewing the complete code. Furthermore, the summary information also enables quick location of code blocks with specific functions during retrieval, facilitating function-based location and querying of the required code blocks, thus further enriching search methods.

[0114] On the other hand, this embodiment also provides another optional implementation: the additional information includes semantic tags. These semantic tags include one or more of the following: function name, parameter list summary, and return type; the semantic tags are used as keywords for matching and retrieval in code block searches.

[0115] Specifically, the above embodiments have already provided the types of code blocks (such as "function", "class", "code segment", "interface", "comment block", etc.) in the metadata fields (additional information) for retrieval filtering and display. In addition, this embodiment also extracts semantic tags such as function names, class names, and method signatures. For example, the metadata of a function block will include the function name, parameter list summary, and return type (if any). These tags can also be used for keyword matching retrieval, further enriching the retrieval methods for code blocks.

[0116] In addition, this embodiment also provides an optional implementation: the additional information also includes the start and end positions of the code block in the source code, as well as the file path of the source code to which it belongs; the start position, end position and file path are used as attribute information to participate in targeted retrieval in the code block retrieval.

[0117] In this embodiment, the metadata of each code block records its start and end positions (line number or character offset) in the source code, as well as the file path of the source code it belongs to (e.g., the source code is stored as a file). This supports targeted retrieval based on files or ranges, while ensuring that the search results can be located back to the original source code file, making it convenient for users to view the complete context.

[0118] On the other hand, as can be seen from the above embodiments, in practical applications, source code can be stored in the form of files (i.e., source files), and this method is not limited to the source code being code from a single source file; it can also be code from multiple source files. Based on this, this embodiment provides an optional implementation: the source code includes code stored in multiple source files; then the code block graph is determined in parallel between different source files; and after the code block graph corresponding to each source file is determined, each code block graph is stored in the code repository, and a correspondence is established between the code block graph and the source files, as well as with the version control system.

[0119] In this embodiment, the code block segmentation method described above can be executed on the source code corresponding to multiple source files. The segmentation of the source code corresponding to multiple source files can be achieved in parallel using multiple threads. Finally, the code block graphs obtained after each method is executed are stored in the code repository for subsequent querying and retrieval. It should be noted that this embodiment is not limited to whether multiple code block graphs are stored independently or integrated into one (or more) code block graphs in the code repository. If stored independently, the code block graphs stored in the code repository are file-level (i.e., source file) graphs. If integrated into one (or more) code block graphs, higher-level code block graphs can be constructed. For example, taking the integration of multiple file-level code block graphs into a higher-level code block graph as an example, a file-level code block graph can be inserted as an independent subgraph into another code block graph to be integrated. The insertion position and insertion relationship can be determined based on the relationship between the source files corresponding to the two code block graphs. Furthermore, this embodiment also establishes a correspondence between the code block graph and the version control system after the code block graph is built. A version control system is used to manage version updates of information within the system. Establishing a correspondence between the code block graph and the version control system means entrusting the version control system with version management. Specifically, a version number can be assigned to the code block graph, and whenever the code block graph is updated, its corresponding version number is also updated accordingly, thereby achieving effective management of the code block graph versions.

[0120] Furthermore, regarding how to implement version updates for the code block graph, this embodiment also provides an optional solution. The above method further includes: S301: When a change is detected in the source file, the changed source file is determined and its syntax is parsed to construct a new abstract syntax tree; S302: The difference between the new and old abstract syntax trees is compared to locate the changed nodes; S303: The corresponding code block subtree for the changed nodes is re-determined, and the code block tree corresponding to the changed source file is updated according to the code block subtree.

[0121] During the evolution of a codebase, source files may be added, deleted, or modified. To maintain consistency between the code block graph and the actual code, this embodiment designs an incremental update mechanism. When a file change is detected, the version control system can re-parse the file to generate a new AST and compare it with the old AST / block tree. By comparing the AST differences, the affected block nodes can be located (e.g., added functions, deleted classes, modified function implementations, etc.). Subsequently, the block partitioning algorithm is re-executed for the changed parts, and the corresponding code block subtree is updated.

[0122] Some common code block update operations include: 1. For newly added code nodes, generate new blocks according to the algorithm and insert them into the corresponding positions in the block tree (update the list of child blocks of the parent node, etc.); 2. For deleted code nodes, remove the corresponding nodes from the code block tree, cascade delete its descendant nodes, and mark the obsolete or deleted records in the metadata index; 3. For modified code nodes, it may be necessary to re-divide: if the modification does not change the structure (such as changes in the internal details of the function but the function signature remains unchanged), the changed child nodes under the node can be further compared; if the modification affects the structure (such as changes in the function signature, or splitting / merging of the original logic), then completely re-divide the blocks with the node as the root and replace the old subtree.

[0123] Based on the refined incremental processing mechanism provided in this embodiment, a full rebuild can be avoided when updating the source code, thus improving maintenance efficiency. In particular, by leveraging the incremental parsing capabilities of AST parsers such as tree-sitter, the incremental AST (diff) can be directly obtained, and then the block tree can be updated in a targeted manner.

[0124] Furthermore, in the maintenance and updating of the code repository, although the maintenance difficulty and efficiency can be reduced and improved through the embodiments provided above, a major principle of code block segmentation is that the segmented code blocks can be restored to the original source code, that is, to ensure the consistency between the updated code repository and the actual source code. Based on this, this embodiment further provides a consistency verification scheme, and the above method further includes: S4: performing consistency verification on the code block tree; if the consistency verification fails, the error is recorded and an alarm is issued.

[0125] The consistency check includes: S401: Traverse all code blocks in the code block tree, concatenate the code blocks, and verify whether the concatenated code is consistent with the source code.

[0126] S402: Check if the list of child blocks of the parent block is consistent with the unique identifier of the code block of its corresponding child block.

[0127] S403: Check if the child block list of the parent block and the unique code block identifier in the parent block identifier of the child block both exist.

[0128] It should be noted that among the above consistency verification steps, step S401 is the most direct and reliable. Based on this, steps S402 and S403 can perform consistency verification from two other aspects. In practical applications, step S401 can be considered a necessary step for consistency verification, while steps S402 and S403 can be implemented as needed.

[0129] In summary, the maintenance mechanisms provided by the above embodiments ensure that the code block tree remains synchronized with the evolution of the codebase and is utilized as a carrier of code knowledge throughout the entire software development lifecycle. Its structured and hierarchical characteristics significantly improve the manageability and usability of code data.

[0130] On the other hand, the code block segmentation method provided in the above embodiments of the present invention ultimately serves the downstream Search Enhancement Generation (RAG) system. Therefore, this embodiment also provides an optional implementation scheme for code block retrieval. After obtaining the code block graph in step S17, the method further includes: S501: storing the code block graph in a vector retrieval database or a full-text retrieval engine.

[0131] In this system, each code block in the code block graph is treated as a file entry; the vector retrieval database and the full-text search engine provide query services to the outside world through the retrieval interface.

[0132] S502: When a code query request is received, the retrieval interface is called to query the corresponding code block.

[0133] Furthermore, as can be seen from the above embodiments, a major advantage of this method is that it can provide rich contextual information (based on parent-child relationships and metadata). This allows for code block segmentation using this method, enabling subsequent retrieval of code blocks from more dimensions and retrieving more diverse information. To this end, this embodiment also provides a suitable implementation scheme: the code query request includes one or more of the following: a standard block query request, a parent block query request, and a child block query request.

[0134] For a standard block query request, step S502 above is specifically S502-A: query the corresponding target code block according to the standard block query request, and return the content text of the target code block as the query result.

[0135] For the parent block query request, the above step S502 is specifically S502-B: query the corresponding target code block according to the parent block query request, and return the content text of the parent block or ancestor block corresponding to the target code block as the query result; wherein, the ancestor block is the code block in the code block graph whose parent-child relationship is above the parent block of the target code block.

[0136] For a sub-block query request, step S502 above is specifically S502-C: query the corresponding target code block according to the sub-block query request, and return the content text of the sub-block or descendant block corresponding to the target code block as the query result; wherein, the descendant block is a code block in the code block graph whose parent-child relationship is located below the target code block.

[0137] Furthermore, regarding how to differentiate the above query requests, this embodiment also provides an optional implementation: configure the above retrieval interface to support a parameter to request a summary of the parent or ancestor block. For example, setting context_level=parent will simultaneously return the summary content of the parent block, helping the model understand the background such as the class / module name in which the block belongs. This method of configuring interface parameters to implement specific query requests is also applicable to the other two types of query requests.

[0138] Furthermore, different query requests can be implemented by setting up multiple retrieval interfaces. Taking the sub-block query request mentioned above as an example, a specific interface (let's say get_children) can be set up. When get_children(class_block_id) is input, it means retrieving a list of all method blocks under that class (class_block_id), and then selecting the required method block content to provide to the downstream model.

[0139] Furthermore, since the above embodiments of the present invention provide a retrieval scheme that can provide rich context corresponding to code blocks, the contextual relationships between various code blocks can be easily understood. In some scenarios, the RAG system may sometimes need to dynamically determine the combination of context fragments provided to the model based on the query. In this case, the retrieval interface needs to allow dynamic assembly of code blocks according to certain rules after the retrieval stage.

[0140] For example, if a user's question involves function A calling function B, the retrieval or inference layer might want to provide both function A and function B as context. Using the previously obtained code block graph, relevant blocks can be easily found by block ID, and multiple block contents can be retrieved at once via an interface, then assembled into the prompt context in the required order. Because all code blocks retain their original code format, the assembled content is still a valid set of code segments that can be directly provided to the generative model.

[0141] To address this requirement, this embodiment also provides an optional solution. If multiple code blocks are retrieved based on the code query request, the method further includes: S503: sorting and assembling the retrieved code blocks according to their position order in the source code to obtain a code segment set, and generating corresponding contextual hint information.

[0142] S504: Returns the set of code snippets and contextual hints as query results.

[0143] Through the interface pattern design provided in the above embodiments, the code block segmentation results of this method can be smoothly integrated into the existing retrieval enhancement generation process. The retrieval end obtains code fragments with reasonable granularity and complete semantics as knowledge sources, improving the performance of code-related tasks; the generation end obtains relevant fragments and context through the interface, significantly reducing the processing pressure of language models on long contexts, while also reducing the occurrence of illusions and misreferences. This interface pattern fully leverages the value of hierarchical code block segmentation.

[0144] Furthermore, by combining the implementation schemes provided in the above embodiments, the following can be obtained: Figure 2 The overall flow of the code block splitting method is shown. For example... Figure 2 As shown, the overall process of the code block segmentation method mainly includes two parts: the code block segmentation part and the code block query part. The code block segmentation part includes: the abstract syntax tree parsing stage, the parent-child abbreviated segmentation stage, the metadata generation stage, and the code block graph construction stage. The code block query part corresponds to the retrieval scheme provided in the above embodiment, supporting querying code blocks according to requirements (summary information) and also supporting querying code blocks and their context based on parent-child relationships.

[0145] It should also be noted that in one possible practical application scenario, code block segmentation is imperceptible to the user and does not require real-time execution (it only occurs when the source code changes). Therefore, it can be implemented offline in the background. Figure 2 As shown, the code block segmentation part can be implemented through an offline preprocessing subsystem. Similarly, the code block query part, which directly interacts with the user and needs to respond to any query requests the user may make in real time, can be implemented through an online query subsystem. There is data interaction between the two subsystems; specifically, after each completion (update) of the code block graph, the offline preprocessing subsystem sends the new code block graph to the online query subsystem to ensure the consistency of the data retrieved by the user.

[0146] In the above embodiments, a code block segmentation method has been described in detail. The present invention also provides an embodiment corresponding to a code block segmentation device. It should be noted that the present invention describes the device embodiment from two perspectives: one based on functional modules, and the other based on hardware.

[0147] From the perspective of functional modules, this embodiment provides a code block segmentation device, such as... Figure 3 As shown, it includes: a syntax parsing module 11, which is used to perform syntax parsing on the source code to construct an abstract syntax tree.

[0148] Density determination module 12 is used to determine the information density of each node in the abstract syntax tree.

[0149] The standard extraction module 13 is used to extract the code corresponding to the node as a standard block if the information density meets the first boundary condition.

[0150] The parent block extraction module 14 is used to mark a node as a parent block node if the information density does not meet the first boundary condition; extract the structural information of the code corresponding to the parent block node, and create a summary block containing the structural information as the parent block.

[0151] The sub-block extraction module 15 is used to determine the child nodes corresponding to the parent block nodes based on the node hierarchy of the abstract syntax tree and the parent-child relationship between nodes, and to extract the code corresponding to the child nodes as sub-blocks.

[0152] The block tree generation module 16 is used to link standard blocks, parent blocks, and child blocks according to the parent-child relationship between nodes in the abstract syntax tree to obtain a code block tree; wherein, child blocks are attached to their corresponding parent blocks; standard blocks, parent blocks, and child blocks all belong to code blocks.

[0153] The graph construction module 17 is used to obtain additional information of each code block in the code block tree and write the additional information as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation. The additional information includes: unique identifier of the code block; for the parent block, the corresponding additional information also includes a list of child blocks, which includes the unique identifier of each child block corresponding to the parent block; for the child block, the corresponding additional information also includes: parent block identifier, which is the unique identifier of the corresponding parent block.

[0154] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0155] In addition to the code block segmentation method embodiment provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instructions, which, when executed by a processor, can implement the steps of the code block segmentation method as described in any of the above embodiments.

[0156] Since the embodiments of the computer program product portion correspond to the embodiments of the method portion, please refer to the description of the embodiments of the method portion for the embodiments of the computer program product portion, which will not be repeated here.

[0157] Figure 4 A structural diagram of an electronic device provided in another embodiment of the present invention, such as... Figure 4As shown, an electronic device includes: a memory 20 for storing a computer program; and a processor 21 for executing the computer program to implement the steps of the code block segmentation method described in the above embodiment. The electronic device provided in this embodiment may include, but is not limited to, a mobile terminal, a personal computer, a workstation, etc.

[0158] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0159] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of a code block segmentation method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, a code block segmentation method.

[0160] In some embodiments, an electronic device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0161] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on an electronic device and may include more or fewer components than shown.

[0162] An electronic device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a code block segmentation method.

[0163] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. A computer program is stored on the non-volatile storage medium, and when executed by a processor, the computer program implements the steps described in the above method embodiments.

[0164] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] The foregoing has provided a detailed description of a code block segmentation method, apparatus, program product, electronic device, and storage medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0166] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A code block segmentation method, characterized in that, include: Perform syntax parsing on the source code to construct an abstract syntax tree; Determine the information density of each node in the abstract syntax tree; If the information density satisfies the first boundary condition, then the code corresponding to the node is extracted as a standard block; If the information density does not meet the first boundary condition, then the node is marked as a parent block node; the structural information of the code corresponding to the parent block node is extracted, and a summary block containing the structural information is created as the parent block; Based on the node hierarchy and parent-child relationships between nodes in the abstract syntax tree, determine the child nodes corresponding to the parent block node, and extract the code corresponding to the child node as a child block; Based on the parent-child relationships between nodes in the abstract syntax tree, the standard block, the parent block, and the child block are linked to obtain a code block tree; wherein, the child block is attached to the corresponding parent block; the standard block, the parent block, and the child block all belong to code blocks; Additional information for each code block in the code block tree is obtained, and this additional information is written into the corresponding code block as metadata to obtain a code block graph for subsequent code block retrieval or generation. The additional information includes: a unique identifier for the code block; for the parent block, the corresponding additional information also includes a list of child blocks, the list of child blocks including the unique identifier of each child block corresponding to the parent block; for each child block, the corresponding additional information also includes: a parent block identifier, the parent block identifier being the unique identifier of the code block corresponding to the parent block.

2. The code block segmentation method according to claim 1, characterized in that, After extracting the code corresponding to the child node as a sub-block, the method further includes: Determine the information density of the sub-block; If the information density of the sub-block still does not meet the first boundary condition, then the node corresponding to the sub-block is marked as the parent node, and the process returns to the step of extracting the structural information of the code corresponding to the parent block node and creating a summary block containing the structural information as the parent block.

3. The code block segmentation method according to claim 1, characterized in that, Determining the information density of each node in the abstract syntax tree includes: By using a depth-first approach, traversing each node in the abstract syntax tree from the root node downwards, the information density of each node is determined sequentially.

4. The code block segmentation method according to claim 1, characterized in that, Determining the information density of a node includes: determining the number of lexical units contained in the code corresponding to the node.

5. The code block segmentation method according to claim 1, characterized in that, Before determining the information density corresponding to each node in the abstract syntax tree, the method further includes: Determine whether the node type of the node belongs to a predefined code block type; If so, proceed to the step of determining the information density of the syntax block corresponding to each node in the abstract syntax tree; If not, skip this node and perform the step of determining whether the node type of the node belongs to a predefined code block type on the child nodes of this node; The code block types include: program structure units, code logic units, documentation and comment units, and special language structure units; The program structure units include: function definitions, class / interface definitions, and module / file-level definitions; The code logic unit includes: main logic paragraphs, anonymous code blocks, and anonymous function expressions; The document and comment unit includes: a document string or block comment attached to a function or class; The special language structure unit includes: specific language structures corresponding to a specific programming language, interface definitions, and enumeration definitions.

6. The code block segmentation method according to claim 5, characterized in that, Also includes: If the information density of any two adjacent code blocks satisfies the second boundary condition and belongs to the same code block type, then the two code blocks are merged. The adjacency relationship includes: sibling adjacency and / or physical adjacency; the sibling adjacency relationship is: the two code blocks are sibling nodes in the code block tree; the physical adjacency relationship is: the two code blocks are consecutive in the source code.

7. The code block segmentation method according to claim 5, characterized in that, When the node is a child node, the code logic unit further includes: compound statement and code block nodes.

8. The code block segmentation method according to claim 5, characterized in that, The additional information also includes: the code block type.

9. The code block segmentation method according to claim 1, characterized in that, The process of parsing the source code to construct an abstract syntax tree includes: Based on the programming language corresponding to the file extension of the source code file, or the conventional programming language, the corresponding abstract syntax tree parser is invoked to perform syntax parsing on the source code.

10. The code block segmentation method according to claim 9, characterized in that, The step of calling the corresponding abstract syntax tree parser to perform syntax parsing on the source code includes: The corresponding block type mapping table is invoked, and the source code is parsed using a unified abstract syntax tree parser. The abstract syntax tree parser uses a pre-abstracted generic block type; the block type mapping table stores the mapping relationship between the block types of the corresponding programming language and the generic block type.

11. The code block segmentation method according to claim 10, characterized in that, Also includes: When adding, deleting, or modifying language support for the abstract syntax tree parser, the corresponding block type mapping table is added, deleted, or modified.

12. The code block segmentation method according to claim 9, characterized in that, Also includes: For code segments in the source code that do not support syntax parsing by the abstract syntax tree parser, they are divided into code blocks based on lines or symbols.

13. The code block segmentation method according to claim 1, characterized in that, The naming rules for uniquely identifying the code block include: The unique identifier of the corresponding code block is determined based on the position of the code block in the code block tree.

14. The code block segmentation method according to claim 13, characterized in that, Also includes: An index directory is established based on the position of the code block in the code block tree for the purpose of retrieving the code block.

15. The code block segmentation method according to claim 1, characterized in that, The additional information also includes: summary information; The summary information includes one or more of the following: the first line comment of the function in the code block, the docstring, the inheritance relationship of the class, and the overall description of the code fragment.

16. The code block segmentation method according to claim 1, characterized in that, The additional information also includes semantic tags; The semantic tags include one or more of the following: function name, parameter list summary, and return type; the semantic tags are used as keywords for matching and retrieval in the code block.

17. The code block segmentation method according to claim 1, characterized in that, The additional information also includes: the start and end positions of the code block in the source code, and the file path of the source code to which it belongs; The start position, the end position, and the file path are used as attribute information in the retrieval of the code block for targeted searching.

18. The code block segmentation method according to claim 1, characterized in that, The source code includes: code stored in multiple source files; The code block graph is then determined in parallel across different source files. Furthermore, after the code block graph corresponding to each source file is determined, each code block graph is stored in the code library, and a correspondence is established between the code block graph and the source file, as well as with the version control system.

19. The code block segmentation method according to claim 18, characterized in that, Also includes: When a change is detected in the source file, the changed source file is parsed to construct a new abstract syntax tree; The two abstract syntax trees, the new and the old, are compared to locate the nodes that have changed. For nodes that have changed, the corresponding code block subtree is redefined, and the code block tree corresponding to the changed source file is updated based on the code block subtree.

20. The code block segmentation method according to claim 1, characterized in that, Also includes: Perform a consistency check on the code block tree. If the consistency check fails, record the error and issue an alarm. The consistency check includes: Traverse all code blocks in the code block tree, concatenate the code blocks, and verify whether the concatenated code is consistent with the source code; Check whether the list of child blocks of the parent block is consistent with the unique identifier of the code block of its corresponding child block; Check whether the unique identifier of the code block exists in the list of child blocks of the parent block and the identifier of the parent block of the child block.

21. The code block segmentation method according to claim 1, characterized in that, After obtaining the code block map, the following is also included: The code block graph is stored in a vector retrieval database or a full-text search engine; each code block in the code block graph is treated as a file entry; the vector retrieval database and the full-text search engine provide query services to the outside world through a retrieval interface; When a code query request is received, the retrieval interface is called to query the corresponding code block.

22. The code block segmentation method according to claim 21, characterized in that, The code query request includes one or more of the following: standard block query request, parent block query request, and child block query request; For the standard block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the standard block query request, and returning the content text of the target code block as the query result; For the parent block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the parent block query request, and returning the content text of the parent block or ancestor block corresponding to the target code block as the query result; wherein, the ancestor block is the code block in the code block graph whose parent-child relationship is above the parent block of the target code block; For the sub-block query request, calling the retrieval interface to query the corresponding code block includes: querying the corresponding target code block according to the sub-block query request, and returning the content text of the sub-block or descendant block corresponding to the target code block as the query result; wherein, the descendant block is the code block in the code block graph whose parent-child relationship is located below the sub-block of the target code block.

23. The code block segmentation method according to claim 21, characterized in that, If multiple code blocks are retrieved according to the code query request, the method further includes: The retrieved code blocks are sorted and assembled according to their position in the source code to obtain a code segment set, and corresponding context prompts are generated. The set of code segments and the context hints are returned as query results.

24. A code block segmentation device, characterized in that, include: The syntax parsing module is used to parse the source code to build an abstract syntax tree; The density determination module is used to determine the information density of each node in the abstract syntax tree; A standard extraction module is used to extract the code corresponding to the node as a standard block if the information density satisfies the first boundary condition. The parent block extraction module is used to mark the node as a parent block node if the information density does not meet the first boundary condition; extract the structural information of the code corresponding to the parent block node, and create a summary block containing the structural information as the parent block; The sub-block extraction module is used to determine the child node corresponding to the parent block node based on the node hierarchy and parent-child relationship between the nodes in the abstract syntax tree, and extract the code corresponding to the child node as a sub-block; The block tree generation module is used to link the standard block, the parent block, and the child block according to the parent-child relationship between nodes in the abstract syntax tree to obtain a code block tree; wherein the child block is attached to the corresponding parent block; the standard block, the parent block, and the child block are all code blocks; The code block graph construction module is used to obtain additional information of each code block in the code block tree and write the additional information as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation. The additional information includes: a unique identifier for the code block; for the parent block, the corresponding additional information also includes a list of child blocks, the list of child blocks including the unique identifier of each child block corresponding to the parent block; for each child block, the corresponding additional information also includes: a parent block identifier, the parent block identifier being the unique identifier of the code block corresponding to the parent block.

25. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the code block segmentation method as described in any one of claims 1 to 23.

26. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the code block segmentation method as described in any one of claims 1 to 23 when executing the computer program.

27. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the code block segmentation method as described in any one of claims 1 to 23.

Images