Code block segmentation method, apparatus, program product, electronic device, and storage medium

Abstract

The application discloses a code block segmentation method and device, a program product, an electronic device and a storage medium, and relates to the technical field of code segmentation. In view of the problem that a traditional code segmentation method is difficult to balance semantic integrity and information density, a code block segmentation method is provided, which is based on an abstract syntax tree and realizes hierarchical segmentation of code through a parent-child abbreviated block strategy. The core idea is to recursively subdivide code blocks according to needs under the premise of maintaining the integrity of code semantic units, and to organize the code blocks in a parent-child relationship. When a code block is too large or contains a complex internal structure, the code block is divided into a plurality of child blocks. The original parent block does not disappear, but is retained in the hierarchical structure as an abstract parent block, the structural information is retained, and the actual implementation details are carried by the child node blocks. The AST provides a structural guide, the parent-child abbreviated block guarantees dynamic adjustment of the division granularity, and finally output data contains code content and hierarchical relationships.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of code splitting, and particularly relates to a code block splitting method, device, program product, electronic device and storage medium. BACKGROUND

[0002] In the management and utilization of large-scale code libraries, how to divide source code into small units suitable for retrieval and understanding is a key problem. Especially in the Code-RAG system, the division of code snippets directly affects the accuracy and efficiency of retrieval. By preprocessing the code library into retrievable snippets, the retrieval-enhanced generation system can quickly locate the relevant context during code generation or question answering, improving the correctness and reliability of the generation results.

[0003] The current mainstream code splitting methods include fixed-length splitting, bracket-based splitting and AST-based splitting. However, these methods either ignore the code structure or lack flexible granularity control, making it difficult to balance semantic integrity and information density. This directly leads to the retrieval system may retrieve irrelevant or incomplete code snippets, affecting the downstream generation effect.

[0004] Therefore, the technical personnel in the field urgently need a code block splitting method to solve the problem that the traditional code splitting method either ignores the code structure or lacks flexible granularity control, making it difficult to balance semantic integrity and information density. SUMMARY

[0005] The purpose of the present application is to provide a code block splitting method, device, program product, electronic device and storage medium to solve the problem that the traditional code splitting method is difficult to balance semantic integrity and information density.

[0006] To solve the above technical problems, the present application provides a code block segmentation method, comprising: performing syntax analysis on source code to construct an abstract syntax tree; determining the information density of each node in the abstract syntax tree; if the information density meets a first boundary condition, extracting the code corresponding to the node as a standard block; if the information density does not meet the first boundary condition, marking the node as a parent block node; extracting the structural information of the code corresponding to the parent block node and creating an abstract block containing the structural information as a parent block; determining the child nodes corresponding to the parent block node according to the node hierarchy of the abstract syntax tree and the parent-child relationship between the nodes, and extracting the code corresponding to the child nodes as child blocks; according to the parent-child relationship between the nodes in the abstract syntax tree, linking the standard blocks, the parent blocks and the child blocks to obtain a code block tree; wherein the child blocks are hung under the corresponding parent blocks; the standard blocks, the parent blocks and the child blocks all belong to code blocks; obtaining additional information of each code block in the code block tree, and writing the additional information as metadata into the corresponding code block to obtain a code block atlas for subsequent code block retrieval or generation; wherein the additional information includes a code block unique identifier; for the parent block, the corresponding additional information further includes a child block list, the child block list including the code block unique identifiers of each child block corresponding to the parent block; for the child block, the corresponding additional information further includes a parent block identifier, which is the code block unique identifier of the corresponding parent block.

[0007] In an optional embodiment, after the step of extracting the code corresponding to the child nodes as child blocks, the method further comprises: determining the information density of the child blocks; if the information density of the child blocks still does not meet the first boundary condition, marking the node corresponding to the child block as a parent node, and returning to the step of extracting the structural information of the code corresponding to the parent block node and creating an abstract block containing the structural information as a parent block.

[0008] In an optional embodiment, the step of determining the information density of each node in the abstract syntax tree comprises: determining the information density of each node in the abstract syntax tree by traversing the nodes from the root node to the bottom in a depth-first manner.

[0009] In an optional embodiment, the step of determining the information density of the node comprises: determining the number of tokens contained in the code corresponding to the node.

[0010] In an alternative embodiment, before the determining the information density corresponding to each node in the abstract syntax tree, the method further comprises: determining whether the node type of the node belongs to a predefined code block type; if yes, proceeding to the step of determining the information density corresponding to the syntax block of each node in the abstract syntax tree; if no, skipping the node and performing the step of determining whether the node type of the node belongs to a predefined code block type on the child node of the node; wherein the code block type comprises: program structure unit, code logic unit, document and comment unit, and special language structure unit; the program structure unit comprises: function definition, class / interface definition, module / file level definition; the code logic unit comprises: main logic paragraph, anonymous code block, anonymous function expression; the document and comment unit comprises: document string or block comment attached before a function or class; the special language structure unit comprises: specific language structure corresponding to a specific programming language, interface definition, enumeration definition.

[0011] In an alternative embodiment, the method further comprises: if the information density of any two code blocks having a neighboring relationship satisfies a second boundary condition and belongs to the same code block type, merging the two code blocks; wherein the neighboring relationship comprises: sibling neighboring relationship and / or physical neighboring relationship; the sibling neighboring relationship is that the two code blocks are sibling nodes in the code block tree; the physical neighboring relationship is that the two code blocks are continuous in the source code.

[0012] In an alternative embodiment, when the node is a child node, the code logic unit further comprises: composite statement and code block node.

[0013] In an alternative embodiment, the additional information further comprises: the code block type.

[0014] In an alternative embodiment, the syntax parsing of the source code to construct the abstract syntax tree comprises: according to the programming language corresponding to the file extension of the source code file, or the conventional programming language, invoking a corresponding abstract syntax tree parser to perform syntax parsing on the source code.

[0015] In an alternative embodiment, the invoking a corresponding abstract syntax tree parser to perform syntax parsing on the source code comprises: invoking a corresponding block type mapping table and performing syntax parsing on the source code by a unified abstract syntax tree parser; wherein the abstract syntax tree parser uses a pre-extracted general block type; the block type mapping table stores the mapping relationship between the block type of the corresponding programming language and the general block type.

[0016] In an optional embodiment, the method further includes: adding, deleting, or modifying the corresponding block type mapping table when adding, deleting, or modifying language support for the abstract syntax tree parser.

[0017] In an optional embodiment, the method further includes: dividing code segments in the source code that do not support syntax parsing by the abstract syntax tree parser into code blocks based on lines or symbols.

[0018] In one optional embodiment, the naming rule for the unique identifier of the code block includes: determining the corresponding unique identifier of the code block based on the position of the code block in the code block tree.

[0019] In an optional embodiment, the method further includes: establishing a corresponding index directory based on the position of the code block in the code block tree for the retrieval of the code block.

[0020] In an optional embodiment, the additional information further includes: summary information; wherein the summary information includes one or more of the following: the first line comment of the function in the code block, docstring, class inheritance relationship, and overall description information of the code fragment.

[0021] In an optional embodiment, the additional information further includes semantic tags; wherein the semantic tags include one or more of function names, parameter list summaries, and return types; the semantic tags are used as keywords for matching and retrieval in the code block retrieval.

[0022] In an optional embodiment, the additional information further includes: the start and end positions of the code block in the source code, and the file path of the source code to which it belongs; the start position, the end position, and the file path are used as attribute information in the retrieval of the code block for targeted retrieval.

[0023] In one optional embodiment, the source code includes: code stored in multiple source files; the code block graph is determined in parallel among different source files; and after the code block graph corresponding to each source file is determined, each code block graph is stored in a code library, and a correspondence is established between the code block graph and the source file, as well as with the version control system.

[0024] In an optional embodiment, the method further includes: when a change is detected in the source file, determining the changed source file for syntax parsing to construct a new abstract syntax tree; performing a difference comparison between the new and old abstract syntax trees to locate the changed nodes; re-determining the corresponding code block subtree for the changed nodes, and updating the code block tree corresponding to the changed source file based on the code block subtree.

[0025] In an alternative embodiment, the method further comprises: performing a consistency check on the code block tree, and if the consistency check fails, recording an error and issuing an alert; wherein the consistency check comprises: traversing all code blocks of the code block tree, and splicing each of the code blocks to verify whether the spliced code is consistent with the source code; checking whether the list of child blocks of the parent block and the code block unique identifier of the corresponding child block are consistent; and checking whether the code block unique identifier exists in the list of child blocks of the parent block and the parent block identifier of the child block.

[0026] In an alternative embodiment, after obtaining the code block graph, the method further comprises: storing the code block graph in a vector retrieval database or a full-text retrieval engine; wherein each of the code blocks in the code block graph is taken as a file entry; the vector retrieval database and the full-text retrieval engine provide query services to the outside through a retrieval interface; and when receiving a code query request, the retrieval interface is called to query the corresponding code block.

[0027] In an alternative embodiment, the code query request comprises one or more of a standard block query request, a parent block query request, and a child block query request; for the standard block query request, the calling of the retrieval interface to query the corresponding code block comprises: querying a target code block corresponding to the standard block query request, and returning the content text of the target code block as a query result; for the parent block query request, the calling of the retrieval interface to query the corresponding code block comprises: querying a target code block corresponding to the parent block query request, and returning the content text of the parent block or ancestor block corresponding to the target code block as a query result; wherein the ancestor block is a code block above the parent block of the target code block in the code block graph; for the child block query request, the calling of the retrieval interface to query the corresponding code block comprises: querying a target code block corresponding to the child block query request, and returning the content text of the child block or descendant block corresponding to the target code block as a query result; wherein the descendant block is a code block below the child block of the target code block in the code block graph.

[0028] In an alternative embodiment, if the code blocks queried according to the code query request are multiple, the method further comprises: sorting and splicing each of the queried code blocks according to the position order in the source code to obtain a code segment set, and generating corresponding context prompt information; and returning the code segment set and the context prompt information as a query result.

[0029] To solve the above technical problems, the application further provides a code block splitting device, comprising: a syntax analysis module, configured to perform syntax analysis on source code to build an abstract syntax tree; a density determination module, configured to determine the information density of each node in the abstract syntax tree; a standard extraction module, configured to extract the code corresponding to the node as a standard block if the information density meets a first boundary condition; a parent block extraction module, configured to mark the node as a parent block node if the information density does not meet the first boundary condition; extract the structural information of the code corresponding to the parent block node, and create an abstract block containing the structural information as a parent block; a child block extraction module, configured to determine the child nodes corresponding to the parent block node according to the node hierarchy and the parent-child relationship between the nodes of the abstract syntax tree, and extract the code corresponding to the child nodes as child blocks; a block tree generation module, configured to link the standard blocks, the parent blocks and the child blocks according to the parent-child relationship between the nodes in the abstract syntax tree, to obtain a code block tree; wherein the child blocks are hung under the corresponding parent blocks; the standard blocks, the parent blocks and the child blocks all belong to code blocks; a graph construction module, configured to obtain additional information of each code block in the code block tree, and write the additional information as metadata into the corresponding code block, to obtain a code block graph for subsequent code block retrieval or generation; wherein the additional information includes a code block unique identifier; for the parent block, the corresponding additional information further includes a child block list, the child block list including the code block unique identifiers of each child block corresponding to the parent block; for the child block, the corresponding additional information further includes a parent block identifier, which is the code block unique identifier of the corresponding parent block.

[0030] To solve the above technical problems, the application further provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the code block splitting method as described above.

[0031] To solve the above technical problems, the application further provides an electronic device, comprising: a memory, configured to store a computer program; a processor, configured to execute the computer program to implement the steps of the code block splitting method as described above.

[0032] To solve the above technical problems, the application further provides a non-volatile storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the code block splitting method as described above.

[0033] The application provides a code block segmentation method, which firstly performs preliminary segmentation on source code in terms of semantics by using an abstract syntax tree (AST), so as to avoid damage to code structure and semantics. Then, the information density of each code block (namely, each node of the AST) obtained after preliminary segmentation is determined. If the information density meets a first boundary condition, it is indicated that the information density of the code block is not too large, and the code block can be directly extracted as a standard code block (namely, a standard block). If the information density does not meet the first boundary condition, it is indicated that the information density of the code block is too large, which may exceed the context window of a downstream model or reduce the retrieval precision of the code block. At this time, the method marks the code block as a parent block (the node is marked as a parent node), and the parent block only retains the structural information of the code, but does not retain specific codes. The structural information can still reflect the overview content and context of the parent block code as a code summary. Retaining no specific codes can avoid the problems caused by the too large information density of a code block. Further, the parent block can be further split by the lower-level child nodes of the parent block in the AST, namely, a large parent block is split into multiple small child blocks. Each child block can contain specific codes, and the child blocks can be spliced together to obtain the complete content of the parent block code, reflect the code details of the parent block, and ensure that the code content is not lost. Moreover, the method adds a code block unique identifier (ID) corresponding to the code block in the code block after splitting the code block. On one hand, the code block ID can facilitate subsequent code block retrieval. On the other hand, the code block IDs of the corresponding child blocks and the parent block are included in each other, so that the parent-child relationship between the child blocks and the parent block can be directly reflected in the code block atlas. In subsequent code block retrieval, if the query target is a parent block with large information density, all child blocks corresponding to the parent block can be returned as the query result based on the parent-child relationship, so as to ensure that the code details of the parent block are not lost. If the query target is a split child block, the corresponding parent block can also be returned based on the parent-child relationship, so as to obtain the overall concept of the class or function corresponding to the child block. That is, the method provides a code block segmentation method based on the AST, but the segmentation granularity is adaptively scaled, so that the advantages of AST segmentation are taken and the inflexible segmentation granularity of AST segmentation is compensated, and a more optimal code block segmentation scheme is realized.

[0034] The code block segmentation device, the computer program product, the electronic device and the nonvolatile storage medium provided by the application correspond to the method and have the same effects. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1A flowchart of a code block segmentation method provided in an embodiment of the present invention.

[0037] Figure 2 This is a hardware implementation architecture diagram of a code block segmentation method provided in an embodiment of the present invention.

[0038] Figure 3 This is a structural diagram of a code block splitting device provided in an embodiment of the present invention.

[0039] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0041] The core of this invention is to provide a code block segmentation method, apparatus, program product, electronic device, and storage medium.

[0042] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] In related technologies, the mainstream code block splitting schemes mainly include fixed-length splitting, brace-based splitting, and abstract syntax tree (AST)-based splitting.

[0044] 1. Fixed-length segmentation: This method divides the code into blocks based on a fixed number of characters or lines. It is simple to implement and widely used in general text processing. However, for structured code, this coarse-grained segmentation often disrupts the code's semantic structure. For example, fixed-size segmentation might truncate a function in the middle, causing the first and second halves of a function to be assigned to different segments. Incomplete code segments are difficult to retrieve correctly and may cause downstream generative models to lose important contextual information, resulting in fragmented semantic units and reduced retrieval and generation quality.

[0045] 2. Based on the curly braces segmentation: in programming languages such as C, C++, Java, etc. using curly braces to define the scope, a common practice is to divide the code block with curly braces as the boundary, for example, the curly brace range of each function or class is taken as a segment. This method preserves the structural integrity of the code to some extent, and avoids breaking up the explicitly defined code block. However, its limitation is that it cannot be directly applied to programming languages without explicit curly braces (such as Python); at the same time, if the code inside a curly brace range is too large (for example, a function contains hundreds of lines of code), taking the entire function as a single segment may still be too large, exceeding the context window of the downstream model or reducing the retrieval precision. In addition, simply cutting according to the curly braces cannot further subdivide the code logic structure, for example, the logical paragraphs within a function are still mixed in one block, which is not conducive to fine retrieval.

[0046] 3. AST-based segmentation: parse the code structure using abstract syntax tree, and divide it by syntax node. For example, each function definition and class definition can be taken as a code segment, or in Python, each module is taken as an independent unit. AST segmentation method can identify the hierarchical structure of the code, and preserve the complete syntax unit, which will not arbitrarily break the function or class like fixed-length segmentation. However, existing AST-based division is usually performed at a pre-defined granularity (such as function level or class level), which has limitations in granularity selection: if the granularity is too coarse, large functions are still difficult to handle; if the granularity is too fine (for example, each statement is taken as a segment), too many fragments will be generated, affecting retrieval efficiency. In addition, the AST structures of different languages are different, and different parsing rules need to be customized when implemented, lacking a universal cross-language solution.

[0047] As can be seen from the above, several mainstream code block segmentation schemes have their own advantages and disadvantages, either ignoring the code structure or lacking flexible granularity control, lacking a code block segmentation method that can balance semantic integrity and information density. To solve this problem, the present application provides a code block segmentation method, as shown in Figure 1 S101: performing syntax analysis on the source code to construct an abstract syntax tree.

[0048] S102: determining the information density of each node in the abstract syntax tree.

[0049] S103: if the information density meets the first boundary condition, extracting the code corresponding to the node as a standard block.

[0050] S104: if the information density does not meet the first boundary condition, marking the node as a parent block node; extracting the structural information of the code corresponding to the parent block node, and creating a summary block containing the structural information as the parent block.

[0051] S105: According to the node hierarchy of the abstract syntax tree and the parent-child relationship between the nodes, determine the child nodes corresponding to the parent block nodes, and extract the code corresponding to the child nodes as the child blocks.

[0052] S106: According to the parent-child relationship between the nodes in the abstract syntax tree, link the standard blocks, parent blocks and child blocks to obtain a code block tree.

[0053] Among them, the child block is hung under the corresponding parent block; the standard block, the parent block and the child block all belong to the code block.

[0054] S107: Obtain the additional information of each code block in the code block tree, and write the additional information as metadata into the corresponding code block to obtain a code block atlas for subsequent code block retrieval or generation.

[0055] Among them, the additional information includes: code block unique identifier (ID); for the parent block, the corresponding additional information further includes a child block list, the child block list includes the code block unique identifier of each child block corresponding to the parent block; for the child block, the corresponding additional information further includes: parent block identifier, the parent block identifier is the code block unique identifier of the corresponding parent block.

[0056] For the above step S101. The specific implementation of the code semantic analysis and splitting by AST can be referred to the above description of the related art, which will not be repeated here. The purpose of step S101 is to preliminarily split the code blocks by AST to obtain preliminary code blocks that retain the hierarchical structure of the code and complete syntax units (one preliminary code block corresponds to one node in AST). Specifically, when splitting the code blocks, the boundary position needs to be determined, which can strictly follow the syntax structure of AST, and the contents belonging to the same node in syntax will not be split. In other words, the block boundary can only be at the AST node boundary, and cannot be at any code position. This ensures that each code block is semantically self-consistent. For example, when a function is split, it can only be split at the statement boundary within the function body, and a compound statement cannot be truncated. By restricting the AST node boundary, this method avoids the semantic damage caused by traditional fixed division. The boundary between blocks is naturally consistent with the structured boundary of the source code, ensuring the structural integrity.

[0057] Then, the core of the method, i.e. steps S102-S105, further split the code blocks based on the parent-child thumbnail block strategy.

[0058] Specifically, step S102 first determines whether the information density of each preliminary code block after the preliminary splitting meets the requirements. The requirement for the information density is to meet the first boundary condition, that is, the first boundary condition should be specifically determined according to the information density requirement of the code block splitting, and the embodiment does not limit this. For example, assuming that the number of lines or characters of the code is taken as the measurement unit of the information density of the code block, the corresponding first boundary condition can be specifically a certain threshold value, and less than or equal to the threshold value is regarded as meeting the first boundary condition, otherwise it is regarded as not meeting the first boundary condition.

[0059] Further, for whether the first boundary condition in step S102 is met, there are two different branches of steps S103 and S104. For branch one, that is, step S103, when the information density of the code block after the preliminary splitting based on the AST meets the requirement, it can be directly extracted as the code block output for subsequent processing. For convenience of distinction, the code block that meets the information density requirement after the preliminary splitting of the AST is referred to as a standard (code) block in this embodiment. For branch two, that is, steps S104 and S105, it is aimed at the code block whose information density does not meet the requirement after the preliminary splitting of the AST. The information density of this type of code block is too large, and if the code is directly extracted, it may exceed the context window of the downstream model or reduce the retrieval accuracy of the code block. Therefore, steps S104 and S105 in the method further split it.

[0060] For step S104, it marks this type of code block (node) as a parent block (parent block node), and only retains the structural information of the corresponding code block in the parent block, without retaining the specific code content. That is, from the perspective of containing content, the parent block is essentially a summary block. However, considering that the complete code of the parent block can also be presented by splicing each child block, the parent block is also regarded as a code block in the subsequent description. It should be noted that the embodiment does not limit the specific content of the structural information, which can include but is not limited to: class name / function name, parameter, document comment, and other information that can help understand the code of the parent block, which essentially belongs to a kind of code summary. The purpose of this step is to avoid the problem of excessive information density of the parent block while retaining the summary of the parent block code, that is, retaining the overall concept of the parent block, without missing any semantic unit.

[0061] Then for step S105. On the premise that the parent block does not retain the specific code, in order to avoid the loss of specific code content, the method splits the parent block into multiple child blocks through step S105, and the child block retains the specific code, thereby embodying the code details of the parent block. Specifically, when the source code is semantically parsed based on the AST, the AST can determine the parent-child relationship between each code block according to the semantic relationship between the codes. Therefore, after the parent block is determined, the code blocks of the child nodes corresponding to the parent block node can be taken as the child blocks corresponding to the parent block based on the parent-child relationship in the AST.

[0062] Based on this, the method realizes further splitting of the parent block with too large information density and not meeting the requirements, and splits the parent block into multiple child blocks with smaller information density. The parent block does not retain specific code content, but only retains code summary. The child block retains specific code content, and the complete parent block code can be obtained after splicing of the child blocks, without losing code content. That is, the parent-child thumbnail block strategy provided by the method can further split the code block (parent block) with too large information density in code details (into multiple child blocks), but still retains the overall concept (the parent block retains the structural information), thereby balancing the retention of code semantics and overall architecture. The split code block structure is much clearer than the fixed cutting piece list, facilitating manual browsing and understanding, and providing context clues for retrieval.

[0063] Further, to clearly represent the relationship between the parent block and the child block, so as to achieve the structural advantage of the split code block mentioned above during retrieval, steps S106 and S107 are further provided in the method. The core of step S106 is to inject additional metadata into the further split code block (standard block, parent block and child block). The additional information (i.e. injected metadata) at least includes the code block ID corresponding to the code block, for uniquely representing the code block. For the parent block and the child block, the additional information further includes the code block ID of the other party, also for representing the parent-child relationship between the code blocks. It should be noted that generally one child block corresponds to one parent block, but one parent block can correspond to multiple child blocks, so the additional information of the parent block contains a child block list (i.e. a set of corresponding child block IDs), and the additional information of the child block contains a parent block ID. Based on the above setting, the corresponding code block can be efficiently and conveniently retrieved through the code block ID during retrieval of the code block. At the same time, when the code block is a parent block, the corresponding child block ID can be found through the child block list in the additional information. If the code details of the parent block need to be queried, the corresponding child blocks can be found through the child block ID, and the complete parent block code can be obtained by splicing the child block code. Similarly, when the code block to be queried is a child block, if the overall concept of the function / class of the child block is to be known, the structural information stored in the corresponding parent block can also be found through the parent block ID in the additional information, and the code summary is returned.

[0064] Further, for step S107, a persistent storage scheme is provided for each divided code block. After the additional information of each code block in the code block tree is written, the parent-child relationship between the code blocks can be more directly and comprehensively represented, and can be regarded as a code block atlas. In the atlas, the nodes represent specific code blocks, and the edges represent the parent-child relationship between the code blocks. The code block atlas shows the hierarchical relationship between the code blocks obtained by switching the source code, and can be used for subsequent retrieval.

[0065] In summary, the present application provides a code block splitting method based on AST but with adaptive granularity scaling. The advantages of AST splitting can be taken into account, and the inflexibility of the granularity of the traditional AST splitting scheme can be compensated for, to implement a parent-child abbreviated block scheme. The parent block at the high level only retains the structural information, i.e., provides the overall concept and context information of the class / function. However, the parent block does not retain the specific code content, which is provided by the child block at the low level. That is, the present method implements hierarchical semantic representation, so that the structure of the split code block is clearer and subsequent retrieval or viewing with flexible granularity scaling is facilitated.

[0066] On the other hand, as known from the above embodiments, for the parent block whose information density does not satisfy the first boundary condition, the problem of excessive information density is solved by splitting the child block. However, in some special scenarios, the split child block may still have the problem of not satisfying the first boundary condition. At this time, the present embodiment also provides a solution accordingly. After step S105 of extracting the code corresponding to the child node as a child block, the above method further includes: S108: determining the information density of the child block; if the information density of the child block still does not satisfy the first boundary condition, marking the node corresponding to the child block as a parent node, and returning to step S104.

[0067] That is, in the present embodiment, a child block can be further divided until all the divided code blocks satisfy the first boundary condition. At this time, it should be noted that the standard block, the parent block and the child block are only identities of the code block, and a code block can have multiple identities. For example, in the present embodiment, all the child blocks should also have the identity of the standard block because they satisfy the first boundary condition. A code block can have both the identity of the parent block and the identity of the child block. For example, in a branch of the code block tree, there are three code blocks from top to bottom according to the parent-child relationship: code block A—code block B—code block C. In this branch, code block B is both the child block of code block A and the parent block of code block C. In addition, code block A can be referred to as the ancestor block of code block C, and code block C can be referred to as the descendant block of code block A.

[0068] It should also be noted that when a code block has both the identity of the parent block and the identity of the child block, the division of the code block is based on the parent block, i.e., only the structural information is retained, and the specific code information is not retained. At this time, the code details represented by the code block can be provided by the child block set of the code block. For example, still taking the above code blocks A, B and C as an example, the code details originally provided by the child block B in the code block A can be further provided by the child block C of the code block B (this example temporarily ignores other child blocks of the code blocks A and B). Similarly, the principle of the three-level parent-child relationship in the above example can also be applied to more levels of parent-child relationship, and the present embodiment will not be described again.

[0069] Based on the implementation provided in the embodiment, more levels of code block division can be further implemented. It can be ensured that each code block can meet the information density requirement, and meanwhile, the loss of any semantic unit of the source code can be avoided.

[0070] On the other hand, after the core part of the method, i.e., the parent-child abbreviated block strategy, is described in detail in the above embodiment, the remaining part of the method is further described in the embodiment. First, the code block operation involved in the steps S101-S105 of the method. The specific code block operation can be implemented by a series of pipeline processing steps and recursive algorithms.

[0071] For example, the embodiment provides an optional implementation for the specific block implementation of step S101. Step S101 specifically includes: determining the information density of each node in the abstract syntax tree by starting from the root node and traversing the nodes in a depth-first manner.

[0072] Further, the root node of the AST is traversed in a depth-first manner in the embodiment. Before entering the child node, it can be determined whether the current node itself constitutes a block and whether the size of the block needs to be further split (i.e., marked as parent block split into child blocks). This strategy of first splitting and then diving ensures that the semantic unit at a higher level is prioritized to form a block, and the boundary judgment will not be disturbed by the nodes at a lower level. For example, a large class will be identified as a block that needs to be split, and then each method in the class will be processed. The class at this level will not be ignored due to the existence of more small nodes in the method. This mechanism of first determining the parent node and then processing the child node enables the method to include as many complete large units as possible in a block, and only split the block when necessary.

[0073] In the embodiment, an optional scheme for preliminary code block division based on the AST is given by using the pipeline + recursion idea. The reason for using the depth-first traversal scheme instead of other optional traversal schemes is that the depth-first traversal scheme can better support the recursive processing of nested structures from top to bottom, and is more suitable for splitting parent-child code blocks. The purpose of using the top-down traversal method from the root node in the embodiment is to try to put as many large nodes as possible into a single code block, so as to reduce the number of divided code blocks and the complexity of the division structure. In addition, it can be known from the above embodiment that if the code block is too large, it can be further divided based on the parent-child abbreviated block strategy. Therefore, the traversal method in the embodiment will not cause the information density of a single code block to be too large. In addition, based on the top-down traversal scheme provided in the embodiment, the bottom-up assembly of the hierarchical structure can be achieved when the child blocks are combined to form the parent block, so that the code block division structure is clearer and more readable.

[0074] On the other hand, an important criterion for the code block division in the above embodiments is whether the information density of the code block satisfies the first boundary condition. For the information density, the above embodiments give a general implementation that the number of lines or non-blank characters in the code block is used as the measure of the information density. However, this scheme may lead to misjudgment due to the different formats used by the source code. Moreover, when the source code uses different programming languages, the comparability of this measure is not strong. In view of this, the present embodiment provides an alternative implementation, and the step S102 of determining the information density of the node specifically includes: determining the number of tokens contained in the code corresponding to the node. The token is the smallest syntax unit in the code, such as keyword, identifier, literal, operator, and separator. This can avoid misjudgment caused by format or blank line and ensure the comparability of the information density under different languages and styles. Further, on the basis of the present embodiment, an alternative scheme of the first boundary condition is that the number of tokens is less than or equal to 512.

[0075] On the other hand, to achieve more general and accurate code block, the present embodiment further provides a further implementation. Before the step S102, the above method further includes: S109: determining whether the node type of the node belongs to a predefined code block type; if yes, go to step S102; if no, go to step S110.

[0076] S110: skip this node and perform step S109 on the child nodes of this node.

[0077] The code block type includes: program structure unit, code logic unit, document and comment unit, and special language structure unit.

[0078] The program structure unit includes: function definition (including method, constructor function, etc.), class / interface definition, module / file level definition; these are usually self-contained logical units in the code, have independent meaning, and are suitable for retrieval granularity. For example, in Java, class and method are the main blocks, in Python, function (def) and class are also blocks, and in C language, function is a block node.

[0079] Code logic units include: main logic paragraph, anonymous code block, anonymous function expression (Lambda expression). It should be noted that this type is an optional type. In some cases, the logic segment inside the function can also be a sub-block, such as the main logic paragraph, anonymous code block, Lambda expression in a large function. If the target function code body is too long, it may need to be further split according to semantic segments. In this case, the statement block nodes in the AST (such as compound statements, code block nodes) can also be included in the block type, but usually only as a sub-block when the parent block needs to be split. That is, the embodiment further provides an optional implementation: when the node is a child node, the code logic unit further includes: compound statements and code block nodes. For example, in the AST representation, an if-else structure or for loop and its internal statements can be classified as a sub-block, which belongs to the function block.

[0080] Document and comment units include: document strings or block comments attached before functions or classes. Specifically, the document string (docstring) or block comment attached before the function or class is closely related to the code unit in semantics. In this embodiment, it is regarded as part of the block content rather than an independent block. However, for the metadata of the code segment, it is recorded whether there is a document description, and if necessary, the comment content is included in the summary to assist in retrieval and understanding. For example, in Python, the document string (docstring) of the function will form a code block together with the function body; the document comment (Javadoc) before the Java method will also be included in the summary information of the method block.

[0081] Special language structure units include: specific language structures, interface definitions, and enumeration definitions corresponding to specific programming languages. Specifically, this type is for specific structures that exist in some languages, such as the impl block (a keyword used to define an implementation block) of Rust (a programming language), the attribute annotation area of C#, etc. If they have independent meanings, they can also be regarded as block nodes. In addition, interface definitions, enumeration definitions, etc. are also classified as structure unit blocks.

[0082] From the above, in order to realize general and accurate code block, the embodiment first defines the type range of the block (node), that is, which node types in the AST will be regarded as units that can be independently blocked. Defining block (node) types needs to consider the common points and unique structures of various languages, and follow the principles of semantic integrity and retrieval usefulness. The definition scheme provided in the above embodiment well meets this requirement.

[0083] Further, based on the above embodiments, the type of each code block can be determined, and the code block type can well reflect whether the semantics of each code block are the same. Based on this, the present embodiment further provides a further implementation scheme, and the above method further comprises: S111: if the information density of any two code blocks having an adjacent relationship satisfies the second boundary condition and belongs to the same code block type, the two code blocks are merged.

[0084] The adjacent relationship comprises a sibling adjacent relationship and / or a physical adjacent relationship. The sibling adjacent relationship is that two code blocks are sibling nodes in the code block tree. The physical adjacent relationship is that two code blocks are continuous in the source code.

[0085] First of all, it needs to be pointed out that the present embodiment also does not limit the specific condition of the second boundary condition. However, it can be known from the above embodiments that, in the present embodiment, the code block that is determined to be "too small" based on the second boundary condition is merged to reduce the number of code blocks. The above first boundary condition is used to determine the "too large" code block. Therefore, in order to enable the two boundary conditions to fully play their own roles, the first boundary condition should include the second boundary condition. Alternatively, for the convenience of understanding, the second boundary condition can be considered to be "smaller" than the first boundary condition. In addition, the code block merging based on the second boundary condition should also avoid the situation that it needs to be split again. That is, after the two code blocks satisfying the second boundary condition are merged, the first boundary condition should also be satisfied, otherwise it will cause the code block to be split again, which wastes resources and may also cause the code block splitting to fall into a dead loop. Based on this, it can be further obtained that the second boundary condition is "smaller" than the first boundary condition.

[0086] For example, in the above embodiment, the information density is measured by the number of tokens, and the first boundary condition is that the number of tokens is less than or equal to 512. An optional second boundary condition can be that the number of tokens is less than or equal to 50. 2*50<512, that is, it satisfies the requirement that the second boundary condition is "smaller" than the first boundary condition by two times.

[0087] It also needs to be pointed out that the present step can use the sibling adjacent relationship of the nodes in the code block tree. Therefore, the step S111 provided in the present embodiment can occur in the process of obtaining the code block tree in step S106, that is, the merging of the adjacent and semantically similar small code blocks can be completed at the same time when the code block tree is generated. It can also occur after step S106, that is, the merging of the adjacent and semantically similar small code blocks can be performed after the code block tree is generated. The present embodiment does not limit this.

[0088] In the present embodiment, a plurality of adjacent small blocks can be merged into a larger block to improve the information density of a single block. For example, a series of continuous very short function definitions can be merged into a module-level code block.

[0089] On the other hand, after the above embodiment gives a specific definition of the code block type, the approximate semantics of each code block can be conveniently and efficiently determined based on the code block type. This effect can also be well applied in the subsequent retrieval process of the code block. In addition, the above embodiment also illustrates that the additional information (metadata) in the present application is not limited to the code block ID, the sub-block list and the parent block ID. Therefore, based on this, the present embodiment further provides an optional implementation scheme for the additional information: the additional information further includes the code block type. That is, in the present embodiment, the code block type is also written into the code block as the metadata of the code block. When subsequent retrieval of the code block is required, the corresponding code block can be retrieved according to the semantic requirement, thereby providing more diverse and flexible retrieval schemes for the retrieval of the code block.

[0090] Further, as can be known from the description of the above related art part, the current common code division scheme based on AST has another significant problem: the AST structures of different languages are different, and different parsing rules often need to be customized when implemented, lacking a universal cross-language scheme. To solve this problem, the present embodiment provides a multi-language adaptation scheme, and the above step S101 is specifically: according to the programming language corresponding to the file extension of the source code file or the agreed programming language, calling a corresponding abstract syntax tree parser to perform syntax analysis on the source code.

[0091] However, if a corresponding AST parser is given for each different programming language, the implementation of the entire scheme becomes difficult and cumbersome. Based on this, the present embodiment further provides a further implementation scheme. The above embodiment step S101: calling a corresponding abstract syntax tree parser to perform syntax analysis on the source code, specifically includes: calling a corresponding block type mapping table and performing syntax analysis on the source code through a unified abstract syntax tree parser. Wherein, the abstract syntax tree parser uses a pre-abstracted general block type; the block type mapping table stores the mapping relationship between the block type of the corresponding programming language and the general block type.

[0092] Specifically, the present embodiment defines the block type under the common framework, and then maps the AST node type of each language. For example, taking one of the general code block types abstracted in the above embodiment, program structure unit, as an example, it includes: module / file, type definition (such as class / interface / enumeration), function / method definition, code segment / logical block. At this time, a block type mapping table can be established for each programming language.

[0093] 1. Java / C++ / C# etc: Map the "class / struct" of AST nodes to the type definition block, "function / method" to the "function" block, and namespace or package as the "module" block. If there is an anonymous inner class inside, it will be treated as a block as needed.

[0094] 2. Python: Module as "module" block, ClassDef as "class" block, and FunctionDef and AsyncFunctionDef as "function" block. Since Python uses indentation and allows functions or classes to be defined inside functions, the recursive algorithm used in the above embodiment naturally handles these nested cases, and the inner definitions are treated as sub-blocks. Top-level flow control (such as if name =='main') is treated as a logical block.

[0095] 3. JavaScript / TypeScript: Top-level functions (function) and classes (class) are defined as blocks. Considering that JavaScript has a functional style and module mechanism, this embodiment can also treat files as "module" blocks. For immediate function expression (IIFE) or Lambda, they can also be included in the logical block processing if needed.

[0096] 4. Other programming languages: For example, Go's func (a keyword used to claim a function), Rust's fn (a keyword used to claim a function) and impl (a keyword used to claim an implementation block), all follow similar rules. In summary, by adjusting the mapping table, new language parsing can be quickly supported without changing the block algorithm flow.

[0097] This embodiment can determine the language to call the corresponding block type mapping table according to the file extension or convention when loading the source code file, and map the special block types of various programming languages to the pre-abstracted general block types. Thus, based on an AST parser implemented based on general block types, semantic parsing of source code of multiple different programming languages can be realized. At the same time, the AST parsing layer of this embodiment shields the differences between programming languages, realizes consistent behavior across languages, and the node traversal and block splitting in subsequent methods do not need to consider the problem of programming languages, greatly simplifying the implementation difficulty and being conducive to improving the execution efficiency.

[0098] On the other hand, as mentioned in the special language structure unit in the above embodiment of code block type classification. Some programming languages have their special structures or habits. Therefore, adjustments can also be made to the block strategy for these programming languages to better adapt to different programming languages and achieve accurate and reliable code block splitting.

[0099] 1. For scripting languages (e.g. Python, JavaScript): Since these programming languages allow execution statements to be written directly at the top level, the top-level statement set can be regarded as a special "module code segment" block to preserve the context of the script execution flow.

[0100] 2. For static languages (e.g. C++, Java): Preprocessing structures such as macros and templates may exist in these programming languages. These preprocessing structures can be expanded as much as possible in the AST parsing stage to obtain a unified AST; if complete expansion is not possible, the macro definitions can be divided into independent blocks according to the structure, or marked as non-parsable blocks for special processing (for details, see the subsequent embodiment of how to handle code segments that do not support AST parsing).

[0101] 3. For multi-paradigm languages (e.g. Scala which supports both object-oriented and functional programming): The block type mapping can include class, object definition and function nodes at the same time. In addition, the closure and Lambda expression of some programming languages are usually expression nodes in the AST, so this method generally does not regard them as top-level blocks, unless their size is large enough to be considered as a sub-block of the function.

[0102] 4. For comments and documents: Different languages have different ways of extracting comments, which should be handled accordingly. For example, the docstring of Python can be obtained as an expression (Expr) node through AST, and the Javadoc of Java needs to be extracted as an additional AST node in the lexical stage, which can be handled separately but provide a unified metadata field in the end.

[0103] In addition, the embodiment further provides a further implementation based on the above embodiment, and the above method further includes: S201: when adding, deleting or modifying the language support of the abstract syntax tree parser, performing adding, deleting or modifying operations on the corresponding block type mapping table. As can be seen from the above, the multi-language support scheme provided based on the above embodiment not only can simplify the implementation difficulty of the overall scheme when supporting multiple languages. And when modifying the supported multiple languages, it can also be more simple and efficient to complete the modification of the supported languages. This embodiment does not need to modify the AST parser, only needs to perform corresponding modification operations on the block type mapping table, that is, to increase, delete or modify the mapping relationship between the general block type, which is easy to implement.

[0104] Based on the multi-language adaptation scheme provided in the above embodiments, the goal of "one framework, general multi-language" is achieved. From the user's perspective, whether the code library contains Java, Python or other languages, it can be retrieved and managed through a unified hierarchical block result. This greatly reduces the difficulty of obtaining context in multi-language projects and improves the generality and practical value of the system.

[0105] On the other hand, as known from the above embodiments, the method still uses AST as the preliminary code block division scheme, aiming to take advantage of the complete code semantics preserved by AST. However, there are some special cases for code block splitting based on AST: 1. There are empty code segments or very small code segments (such as empty functions, empty classes, or blocks containing only a small number of simple statements); 2. There are code segments that do not support AST parsing (such as code that cannot be parsed or binary macro expansion parts). Among them, the above special case 1 can be solved by the adjacent code block merging scheme provided in the foregoing embodiments.

[0106] As for the above special case 2, the present embodiment also provides an optional scheme accordingly, and the above method further comprises: S202: For code segments in the source code that do not support syntax parsing by the abstract syntax tree parser, divide the code segments into code blocks according to lines or symbols. It should be noted that the present embodiment provides a fallback strategy for code segments that do not support AST. That is, the boundaries can be approximately determined by lines or symbols to isolate the unparseable part into a separate block as much as possible to ensure that it does not affect the division of other code parts.

[0107] Comprehensive scheme provided in the present embodiment and the above embodiments, the method can accurately identify each suitable boundary point when dividing code blocks: neither missing internal units that should be divided, nor forcibly breaking down contents that should not be divided. The generated code block boundaries are clear and reasonable, each block corresponds to a complete paragraph or structure in the source code, and has a clear start and end range and context meaning.

[0108] On the other hand, the above embodiments explain that the code block ID can be used to uniquely determine each code block and to determine the parent-child relationship between code blocks. However, it does not strictly limit how the code block ID is generated, as long as it meets the unique correspondence. Based on this, the present embodiment provides an optional code block ID naming rule: determining the corresponding code block unique identifier according to the position of the code block in the code block tree.

[0109] That is, the embodiment uses the position of the code block in the code block tree to provide a hierarchical coding scheme. As known from the above embodiment, the code block types include module or file level definitions, class or interface level definitions, and function or method level definitions. The code block sizes corresponding to these different code block types are not necessarily the same. For example, a possible code block has a hierarchical coding of 1.2.3 according to the naming rules, which represents the third method block of the second class under the first file. The ID can intuitively reflect the position of the code block in the code block tree, facilitating quick positioning of parent nodes and child nodes.

[0110] Further, the embodiment further provides a further implementation based on the above embodiment, and the method further includes: S203: establishing a corresponding index directory according to the position of the code block in the code block tree, for retrieval of the code block.

[0111] Specifically, the embodiment can store the corresponding hierarchical index according to the position hierarchy of the code block, that is, establish an index directory level by level according to files / modules, classes, and functions. For example, in a database, the code block ID can be first indexed according to the file ID, then according to the class ID under the file, and finally according to the function ID in the lower layer. Such a storage structure naturally corresponds to the code block tree, and can efficiently perform hierarchical queries. Further, the code block ID as a kind of metadata can also be designed to have a unified data interface for querying, and the metadata can be provided to the outside through an object model or JSON / XML (both belong to data exchange formats). In addition, when implementing the index in actual applications, full-text indexes can also be established for commonly used retrieval fields (such as name, summary, docstring, etc.), to support quick retrieval of related code block nodes through keywords or symbol names.

[0112] In addition, the metadata (additional information) of the code block can also be increased with other information as needed in addition to the several examples given above. For example, the embodiment also provides an optional implementation: the additional information further includes summary information; wherein the summary information includes one or more of the first sentence of the function in the code block, the document string, the inheritance relationship of the class, and the overall description information of the code snippet.

[0113] The purpose of adding summary information in the additional information in the embodiment is to understand the general content of the code block without viewing the complete code. In addition, the summary information also enables quick positioning of the code block of a specific function during retrieval, facilitating function-based positioning and querying of the required code block, and further enriching the query means.

[0114] In another aspect, the embodiment also provides another optional implementation: the additional information further includes a semantic tag. The semantic tag includes one or more of a function name, a parameter list summary, and a return type; and the semantic tag is used for keyword matching search in the search of the code block.

[0115] Specifically, in the above embodiment, the type of the code block (e.g., "function", "class", "code segment", "interface", "comment block", etc.) is included in the metadata field (additional information) for search filtering and display. In addition, the embodiment extracts the function name, class name, method signature, etc. as semantic tags. For example, the metadata of a function block includes the function name, parameter list summary, and return type (if any). These tags can also be used for keyword matching search, further enriching the search means of the code block.

[0116] In addition, the embodiment also provides an optional implementation: the additional information further includes the starting position and the ending position of the code block in the source code, and the file path of the source code; and the starting position, the ending position, and the file path are used as attribute information for directional search in the search of the code block.

[0117] In the embodiment, the metadata of each code block records the starting position and the ending position (line number or character offset) of the code block in the source code, and the file path of the source code (e.g., the source code is stored in the form of a file). In this way, directional search according to files or ranges can be supported, and the search result can be located back to the original source code file, facilitating the user to view the complete context.

[0118] On the other hand, as known from the above embodiment, in actual application, the source code can be stored in the form of a file (i.e., a source file), and the method is not limited to the source code being the code in a certain source file, but can also be the code in multiple source files. Based on this, the embodiment provides an optional implementation: the source code includes the code stored in multiple source files; the determination of the code block atlas is performed in parallel between different source files; and after the code block atlas corresponding to each source file is determined, the code block atlas is stored in the code library, and the corresponding relationship between the code block atlas and the source file, and the version control system is established.

[0119] That is, in the embodiment, the code block splitting method of the above embodiment is supported to be respectively executed on the source code corresponding to the plurality of source files. The splitting of the source code corresponding to the plurality of source files can be implemented in parallel through a plurality of threads, and finally the code block atlas obtained after each method is executed is stored in the code library for subsequent query and calling. It should be noted that the embodiment is not limited to whether the plurality of code block atlases are independently stored or integrated into one (or more) code block atlas in the code library. If it is independently stored, the code block atlas stored in the code library is a file-level (i.e., source file) atlas. If it is integrated into one (or more) code block atlas, a higher-level code block atlas can be constructed. For example, a plurality of file-level code block atlases are integrated into a higher-level code block atlas. One file-level code block atlas can be inserted into another code block atlas to be integrated as an independent subgraph. The insertion position and insertion relationship can be determined according to the relationship between the source files corresponding to the two code block atlases. In addition, the embodiment also establishes the corresponding relationship between the code block atlas and the version control system after the code block atlas is established. The version control system is a system for managing version updates of information in the system. The establishment of the corresponding relationship between the version control system is to hand over the code block atlas to the version control system for version management. Specifically, a version number can be assigned to the code block atlas, and the corresponding version number is updated when the code block atlas is updated, thereby realizing effective management of the version of the code block atlas.

[0120] Further, for how the code block atlas realizes version update, the embodiment also provides an optional scheme, and the above method further includes: S301: when detecting that a source file changes, determining the changed source file to perform syntax analysis to construct a new abstract syntax tree; S302: comparing the new and old abstract syntax trees to locate the changed node; S303: re-determining the corresponding code block subtree of the changed node, and updating the code block tree corresponding to the changed source file according to the code block subtree.

[0121] The code library will have new source files added, deleted or modified in the evolution process. In order to maintain the consistency of the code block atlas and the actual code, the embodiment designs an incremental update mechanism. When detecting that a file changes, the version control system can generate a new AST by re-parsing the file, and compare it with the old AST / block tree. By comparing the AST difference, the affected block node (such as adding a function, deleting a class, modifying the internal implementation of a function, etc.) can be located. Then, the block algorithm is re-executed for the changed part, and the corresponding code block subtree is updated.

[0122] Some common code block update operations include: 1. For a newly added code node, a new block is generated according to the algorithm and inserted into the corresponding position of the block tree (update the child block list of the parent node, etc.); 2. For a deleted code node, the corresponding node in the code block tree is removed, and its descendants are deleted in cascade, and the metadata index is marked as abandoned or deleted record; 3. For a modified code node, it may need to be re-divided: if the modification does not change the structure (such as the details of the function are changed but the function signature is not changed), the changed child nodes under the node can be further compared; if the modification affects the structure (such as the function signature is changed, or the original logic is split / combined), the old subtree is replaced by a new block tree with the node as the root.

[0123] Based on the fine incremental processing mechanism provided in the embodiment, full reconstruction can be avoided when the source code is updated, and the maintenance efficiency is improved. In particular, with the incremental parsing capability of the AST parser such as tree-sitter, the AST increment (diff) can be directly obtained, and then the block tree is updated accordingly.

[0124] Further, in the maintenance and update of the code library, although the maintenance difficulty can be reduced and the maintenance efficiency can be improved by the above-mentioned embodiments of the present application, one of the principles of code block division is that the divided code blocks can restore the original source code, that is, to ensure the consistency between the updated code library and the actual source code. Based on this, the embodiment further provides a consistency verification scheme, and the above-mentioned method further comprises: S4: performing consistency check on the code block tree, and if the consistency check fails, recording the error and alarming.

[0125] The consistency check comprises: S401: traversing all code blocks of the code block tree, and splicing each code block to verify whether the spliced code is consistent with the source code.

[0126] S402: checking whether the child block list of the parent block and the code block unique identifier of the corresponding child block are consistent.

[0127] S403: checking whether the code block unique identifiers in the child block list of the parent block and the parent block identifier of the child block exist.

[0128] It should be noted that in the above-mentioned consistency checking steps, step S401 is the most direct and reliable consistency checking step. On this basis, steps S402 and S403 can perform consistency checking from two other aspects. In actual application, step S401 can be used as a necessary item for consistency checking, and steps S402 and S403 can be selected according to actual needs.

[0129] In summary, the series of maintenance mechanisms provided by the above embodiments enable the code block tree to keep pace with the evolution of the code base and be utilized as a carrier of code knowledge throughout the software development life cycle. The structured and hierarchical nature of the code block tree greatly improves the manageability and usability of the code data.

[0130] In another aspect, the code block segmentation method provided by the above embodiments ultimately serves the downstream retrieval augmentation generation (RAG) system. Therefore, the present embodiment also provides an optional implementation for the retrieval of code blocks. After obtaining the code block graph in step S17, the method further comprises: S501: storing the code block graph in a vector retrieval database or a full-text retrieval engine.

[0131] In the code block graph, each code block is treated as a file entry; the vector retrieval database and the full-text retrieval engine provide query services to the outside through a retrieval interface.

[0132] S502: When a code query request is received, the retrieval interface is called to query the corresponding code block.

[0133] Further, as known from the above embodiments, one of the great advantages of the present method is that it can provide rich contextual information (based on parent-child relationships and metadata). After the code block is segmented by the present method, the subsequent retrieval can be implemented from more dimensions of requirements, and more diverse information can be retrieved. In view of this, the present embodiment also provides a corresponding implementation: the code query request comprises one or more of a standard block query request, a parent block query request, and a child block query request.

[0134] For the standard block query request, step S502 is specifically S502-A: querying the corresponding target code block according to the standard block query request, and returning the content text of the target code block as the query result.

[0135] For the parent block query request, step S502 is specifically S502-B: querying the corresponding target code block according to the parent block query request, and returning the content text of the parent block or ancestor block corresponding to the target code block as the query result; wherein the ancestor block is a code block in the code block graph that is located above the parent block of the target code block in the parent-child relationship.

[0136] For the child block query request, step S502 is specifically S502-C: querying the corresponding target code block according to the child block query request, and returning the content text of the child block or descendant block corresponding to the target code block as the query result; wherein the descendant block is a code block in the code block graph that is located below the child block of the target code block in the parent-child relationship.

[0137] Further, for how to distinguish the above query request, the embodiment also provides an optional implementation: the retrieval interface is configured to support a parameter to request a parent or ancestor block profile. For example, setting context_level=parent returns the summary content of the parent block at the same time, helping the model to understand the class / module name and other background of the block. This way of achieving specific query requests by configuring interface parameters is also applicable to the other two query requests.

[0138] In addition, different query requests can also be achieved by additionally setting multiple retrieval interfaces. Taking the above sub-block query request as an example, a specific interface (assuming get_children) can be set. When get_children (class_block_id) is input, it represents obtaining the list of all method blocks under the class (class_block_id), and then selecting the required method block content to provide to the downstream model.

[0139] Further, since the above embodiment of the present application provides a retrieval scheme capable of providing rich context corresponding to code blocks, the context relationship between various code blocks can be easily known. In some scenarios, the RAG system sometimes needs to dynamically determine the combination of context fragments provided to the model according to the query. At this time, the retrieval interface needs to allow dynamic assembly of code blocks after the retrieval stage according to certain rules.

[0140] For example, if the user's question involves function A calling function B, the retrieval or reasoning layer may want to provide both function A and function B as context. Using the previously obtained code block atlas, the relevant blocks can be easily found by block ID, and multiple block contents can be obtained at once through the interface, and then assembled into a prompt context in the required order. Because all code blocks retain the original code format, the assembled content is still a set of legal code segments, which can be directly provided to the generation model.

[0141] To meet this demand, the embodiment also adaptively provides an optional scheme. If the code blocks queried according to the code query request are multiple, the method further includes: S503: sorting and assembling the queried code blocks according to their position order in the source code to obtain a code segment set, and generating corresponding context prompt information.

[0142] S504: returning the code segment set and the context prompt information as the query result.

[0143] Through the interface mode design provided by the above embodiments, the code block segmentation result of the method can be smoothly integrated into the existing retrieval enhancement generation process. The retrieval end obtains code fragments with reasonable granularity and complete semantics as a knowledge source, thereby improving the performance of code-related tasks. The generation end obtains related fragments and context through the interface, thereby greatly reducing the processing pressure of the language model on long context and reducing the occurrence of hallucinations and false references. This interface mode fully plays the value of hierarchical code block division.

[0144] In addition, according to the embodiments provided in the above embodiments, the overall flow of the code block segmentation method can be obtained as shown in Figure 2 As shown in Figure 2 The overall flow of the code block segmentation method mainly includes two parts: a code block segmentation part and a code block query part. The code block segmentation part includes an abstract syntax tree parsing stage, a parent-child thumbnail block stage, a metadata generation stage, and a code block atlas construction stage. The code block query part corresponds to the retrieval scheme provided in the above embodiments, supports querying code blocks according to requirements (abstract information), and also supports querying code blocks and their context according to parent-child relationships.

[0145] It should be further noted that in a possible actual application scenario, the code block segmentation is imperceptible to the user and does not require real-time performance (only when the source code changes will it be performed), so it can be implemented offline in the background. Therefore, as shown in Figure 2 The code block segmentation part can be implemented through an offline preprocessing subsystem. Similarly, the code block query part is the part that directly interacts with the user and needs to respond to the query request that the user may issue in real time, and can be implemented through an online query subsystem. There is data interaction between the two subsystems, and specifically, the offline preprocessing subsystem sends the new code block atlas to the online query subsystem after completing (updating) the code block atlas each time, to ensure the consistency of the data queried by the user.

[0146] In the above embodiments, a code block segmentation method is described in detail, and the present application also provides an embodiment of a code block segmentation device. It should be noted that the embodiment of the device part is described from two angles, one based on functional modules and the other based on hardware.

[0147] Based on the functional module angle, the embodiment provides a code block segmentation device, as shown in Figure 3 which includes a syntax analysis module 11 for syntax analysis of source code to construct an abstract syntax tree.

[0148] A density determination module 12 is configured to determine the information density of each node in the abstract syntax tree.

[0149] The standard extraction module 13 is configured to extract the code corresponding to the node as a standard block if the information density satisfies the first boundary condition.

[0150] The parent block extraction module 14 is configured to mark the node as a parent block node if the information density does not satisfy the first boundary condition, extract the structural information of the code corresponding to the parent block node, and create an abstract block containing the structural information as the parent block.

[0151] The child block extraction module 15 is configured to determine the child nodes corresponding to the parent block node according to the node hierarchy and the parent-child relationship between the nodes of the abstract syntax tree, and extract the code corresponding to the child nodes as child blocks.

[0152] The block tree generation module 16 is configured to link the standard blocks, the parent blocks and the child blocks according to the parent-child relationship between the nodes of the abstract syntax tree, to obtain a code block tree; wherein the child blocks are hung under the corresponding parent blocks; the standard blocks, the parent blocks and the child blocks all belong to code blocks.

[0153] The atlas construction module 17 is configured to obtain additional information of each code block in the code block tree, and write the additional information as metadata into the corresponding code block, to obtain a code block atlas for subsequent code block retrieval or generation; wherein the additional information includes a code block unique identifier; for the parent block, the corresponding additional information further includes a child block list, the child block list including the code block unique identifiers of the child blocks corresponding to the parent block; for the child block, the corresponding additional information further includes a parent block identifier, the parent block identifier being the code block unique identifier of the corresponding parent block.

[0154] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and are not described here.

[0155] In addition to the embodiments of the code block segmentation method provided in the above embodiments, the present application also provides an embodiment corresponding to a computer program product. The computer program product includes computer programs / instructions, which, when executed by a processor, can implement the steps of the code block segmentation method according to any one of the above embodiments.

[0156] Since the embodiments of the computer program product part correspond to the embodiments of the method part, the embodiments of the computer program product part are described in the description of the embodiments of the method part, and are not described here.

[0157] Figure 4 A structural diagram of an electronic device according to another embodiment of the present application is shown in FIG. 4. Figure 4As shown, an electronic device includes a memory 20 for storing a computer program, and a processor 21 for implementing steps of a code block splitting method as described above when executing the computer program. The electronic device provided by the embodiment can include, but is not limited to, a mobile terminal, a personal computer, a workstation, etc.

[0158] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA). The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a Central Processing Unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a Graphics Processing Unit (GPU) for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 21 can further include an Artificial Intelligence (AI) processor for processing machine learning related computing operations.

[0159] The memory 20 can include one or more computer-readable storage media, which can be non-transitory. The memory 20 can further include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In the embodiment, the memory 20 is at least used to store a computer program 201, wherein the computer program is loaded and executed by the processor 21, and can implement the steps of the code block splitting method disclosed in any of the preceding embodiments. In addition, the resources stored in the memory 20 can further include an operating system 202 and data 203, etc., and the storage mode can be temporary storage or permanent storage. The operating system 202 can include Windows, Unix, Linux, etc. The data 203 can include, but is not limited to, a code block splitting method, etc.

[0160] In some embodiments, the electronic device can further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0161] Those skilled in the art can understand that,Figure 4 The structure shown in the figures does not constitute a limitation on an electronic device, and can include more or fewer components than shown.

[0162] An electronic device provided by an embodiment of the present application includes a memory and a processor. The processor, when executing a program stored in the memory, can implement a code block splitting method.

[0163] Finally, the present application also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program. When the computer program is executed by a processor, the steps described in the above method embodiment are implemented.

[0164] It can be understood that if the method in the above embodiment is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and executes all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0165] The above provides a code block splitting method, device, program product, electronic device and storage medium. In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts are described in the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified. These improvements and modifications also fall within the scope of the present application.

[0166] It also needs to be explained that in the present specification, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A code block segmentation method, characterized by, The method comprises the following steps: parsing source code to construct an abstract syntax tree; determining the information density of each node in the abstract syntax tree; if the information density meets a first boundary condition, extracting the code corresponding to the node as a standard block; if the information density does not meet the first boundary condition, marking the node as a parent block node, extracting the structural information of the code corresponding to the parent block node, and creating an abstract block containing the structural information as a parent block; determining the child nodes corresponding to the parent block node according to the node hierarchy and parent-child relationship between nodes of the abstract syntax tree, and extracting the code corresponding to the child nodes as child blocks; linking the standard blocks, the parent blocks and the child blocks according to the parent-child relationship between nodes in the abstract syntax tree to obtain a code block tree; wherein the child blocks are hung under the corresponding parent blocks; the standard blocks, the parent blocks and the child blocks all belong to code blocks; obtaining additional information of each code block in the code block tree, and writing the additional information as metadata into the corresponding code block to obtain a code block atlas for subsequent code block retrieval or generation; wherein the additional information includes a code block unique identifier; for the parent block, the corresponding additional information further includes a child block list, which includes the code block unique identifiers of each child block corresponding to the parent block; for the child block, the corresponding additional information further includes a parent block identifier, which is the code block unique identifier of the corresponding parent block.

2. The code block segmentation method of claim 1, wherein, After the step of extracting the code corresponding to the child nodes as child blocks, the method further comprises the following steps: determining the information density of the child blocks; if the information density of the child blocks still does not meet the first boundary condition, marking the node corresponding to the child block as a parent node, and returning to the step of extracting the structural information of the code corresponding to the parent block node and creating an abstract block containing the structural information as a parent block.

3. The code block segmentation method of claim 1, wherein, The step of determining the information density of each node in the abstract syntax tree comprises the following steps: traversing each node in the abstract syntax tree from the root node in a depth-first manner to determine the information density of each node in sequence.

4. The code block segmentation method of claim 1, wherein, The step of determining the information density of the node comprises the following step:

5. The code block segmentation method of claim 1, wherein, determining the number of tokens contained in the code corresponding to the node. Before the step of determining the information density of each node in the abstract syntax tree, the method further comprises the following steps: determining whether the node type of the node belongs to a predefined code block type; if yes, proceeding to the step of determining the information density of each node in the abstract syntax tree; if no, skipping the node and performing the step of determining whether the node type of the node belongs to a predefined code block type on the child nodes of the node; wherein the code block type includes a program structure unit, a code logic unit, a document and comment unit, and a special language structure unit; the program structure unit includes a function definition, a class / interface definition, and a module / file level definition; the code logic unit includes a main logic paragraph, an anonymous code block, and an anonymous function expression. The document and comment unit includes a document string or a block comment attached to a function or a class; The special language structure unit includes a special language structure, an interface definition, and an enumeration definition corresponding to a specific programming language.

6. The code block segmentation method of claim 5, wherein, Further comprising: If the information density of any two code blocks with an adjacent relationship satisfies a second boundary condition and belongs to the same code block type, the two code blocks are merged; The adjacent relationship includes a sibling adjacent relationship and / or a physical adjacent relationship; the sibling adjacent relationship is that the two code blocks are sibling nodes in the code block tree; the physical adjacent relationship is that the two code blocks are continuous in the source code.

7. The code block segmentation method of claim 5, wherein, When the node is a child node, the code logic unit further includes a compound statement and a code block node.

8. The code block segmentation method of claim 5, wherein, The additional information further includes the code block type.

9. The code block segmentation method of claim 1, wherein, The syntax analysis of the source code to construct an abstract syntax tree includes: According to the programming language corresponding to the file extension of the source code file or the conventional programming language, a corresponding abstract syntax tree parser is called to perform syntax analysis on the source code.

10. The code block segmentation method of claim 9, wherein, The calling of the corresponding abstract syntax tree parser to perform syntax analysis on the source code includes: A corresponding block type mapping table is called, and a unified abstract syntax tree parser is used to perform syntax analysis on the source code; The abstract syntax tree parser uses a pre-extracted general block type; the block type mapping table stores the mapping relationship between the block type of the corresponding programming language and the general block type.

11. The code block segmentation method of claim 10, wherein, Further comprising: When the language support of the abstract syntax tree parser is added, deleted, or modified, the corresponding block type mapping table is added, deleted, or modified.

12. The code block segmentation method of claim 9, wherein, Further comprising: For a code segment in the source code that does not support syntax analysis by the abstract syntax tree parser, the code segment is divided into code blocks according to lines or symbols.

13. The code block segmentation method of claim 1, wherein, The naming rule of the code block unique identifier includes: The corresponding code block unique identifier is determined according to the position of the code block in the code block tree.

14. The code block segmentation method of claim 13, wherein, Further comprising: An index directory corresponding to the position of the code block in the code block tree is established for retrieval of the code block.

15. The code block segmentation method of claim 1, wherein, The additional information further includes summary information; The summary information includes one or more of the first sentence comment of a function in the code block, the document string, the inheritance relationship of a class, and the overall description information of a code segment.

16. The code block segmentation method of claim 1, wherein, The additional information further includes a semantic tag; The semantic tag includes one or more of a function name, a parameter list summary, and a return type; the semantic tag is used as a keyword for matching retrieval in retrieval of the code block.

17. The code block segmentation method of claim 1, wherein, The additional information further includes the starting position and the ending position of the code block in the source code, and the file path of the source code; The starting position, the ending position, and the file path are used as attribute information for directional retrieval in retrieval of the code block.

18. The code block segmentation method of claim 1, wherein, The source code includes code stored in a plurality of source files; The determination of the code block atlas is performed in parallel between different source files. and after determining the code block graph corresponding to each of the source files, storing each of the code block graphs in a code library and establishing a corresponding relationship between the code block graph and the source file and a version control system.

19. The code block segmentation method of claim 18, wherein, Further comprising: when detecting a change in the source file, determining the changed source file to perform syntax analysis to construct a new abstract syntax tree; performing a differential comparison between the new and old abstract syntax trees to locate the changed node; re-determining the corresponding code block subtree for the changed node and updating the code block tree corresponding to the changed source file according to the code block subtree.

20. The code block segmentation method of claim 1, wherein, Further comprising: performing a consistency check on the code block tree, and if the consistency check fails, recording an error and issuing an alarm; wherein the consistency check comprises: traversing all code blocks of the code block tree and concatenating each of the code blocks to verify whether the concatenated code is consistent with the source code; checking whether the code block unique identifier of the child block list of the parent block and the corresponding child block are consistent; checking whether the code block unique identifier of the child block list of the parent block and the parent block identifier of the child block are both present.

21. The code block segmentation method of claim 1, wherein, After obtaining the code block graph, further comprising: storing the code block graph in a vector retrieval database or a full-text retrieval engine; wherein each of the code blocks in the code block graph is taken as a file entry; the vector retrieval database and the full-text retrieval engine provide query services to the outside through a retrieval interface; when receiving a code query request, calling the retrieval interface to query the corresponding code block.

22. The code block segmentation method of claim 21, wherein, The code query request includes one or more of a standard block query request, a parent block query request, and a child block query request; for the standard block query request, the calling the retrieval interface to query the corresponding code block includes querying the corresponding target code block according to the standard block query request, and returning the content text of the target code block as the query result; for the parent block query request, the calling the retrieval interface to query the corresponding code block includes querying the corresponding target code block according to the parent block query request, and returning the content text of the parent block or ancestor block corresponding to the target code block as the query result; wherein the ancestor block is the code block above the parent block of the target code block in the code block graph; for the child block query request, the calling the retrieval interface to query the corresponding code block includes querying the corresponding target code block according to the child block query request, and returning the content text of the child block or descendant block corresponding to the target code block as the query result; wherein the descendant block is the code block below the child block of the target code block in the code block graph.

23. The code block segmentation method of claim 21, wherein, If the code block queried according to the code query request is multiple, the method further comprises: sorting and assembling each of the code blocks queried according to the position order in the source code to obtain a code segment set, and generating corresponding context prompt information; Return the code segment set and the context prompt information as a query result.

24. A code block segmentation apparatus characterized by comprising: The method comprises the following steps: A syntax analysis module is configured to perform syntax analysis on source code to construct an abstract syntax tree; A density determination module is configured to determine the information density of each node in the abstract syntax tree; A standard extraction module is configured to extract the code corresponding to the node as a standard block if the information density meets a first boundary condition; A parent block extraction module is configured to mark the node as a parent block node if the information density does not meet the first boundary condition, extract the structural information of the code corresponding to the parent block node, and create an abstract block containing the structural information as a parent block; A child block extraction module is configured to determine the child nodes corresponding to the parent block node according to the node hierarchy and the parent-child relationship between the nodes of the abstract syntax tree, and extract the code corresponding to the child nodes as child blocks; A block tree generation module is configured to link the standard blocks, the parent blocks and the child blocks according to the parent-child relationship between the nodes of the abstract syntax tree to obtain a code block tree; wherein the child blocks are hung under the corresponding parent blocks; the standard blocks, the parent blocks and the child blocks all belong to code blocks; A graph construction module is configured to obtain additional information of each code block in the code block tree, and write the additional information as metadata into the corresponding code block to obtain a code block graph for subsequent code block retrieval or generation; wherein the additional information includes a code block unique identifier; for the parent block, the corresponding additional information further includes a child block list, and the child block list includes the code block unique identifiers of each child block corresponding to the parent block; for the child block, the corresponding additional information further includes a parent block identifier, which is the code block unique identifier of the corresponding parent block.

25. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the code block segmentation method according to any one of claims 1 to 23.

26. An electronic device, comprising: The method comprises the following steps: A memory is configured to store a computer program; A processor is configured to execute the computer program to implement the steps of the code block segmentation method according to any one of claims 1 to 23.

27. A non-volatile storage medium, comprising: The non-volatile storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the code block segmentation method according to any one of claims 1 to 23.

Images