Warehouse-level full-quantity code auditing method and device

Abstract

The invention discloses a warehouse-level full-quantity code auditing method and device, and the method comprises the following steps: downloading a target warehouse from a code hosting platform to the local; scanning and identifying source code files, segmenting each file into a plurality of code fragments based on an abstract syntax tree, and generating metadata containing information such as paths, languages, structures and the like; converting the code snippets into vector representations through a vector model, and storing the vector representations into a vector database; constructing an examination chain and a summary chain based on a large language model, wherein the examination chain and the summary chain are respectively used for multi-dimensionally examining a single fragment and summarizing to generate a warehouse-level report; during auditing, establishing a symbol index and calling relation graph based on metadata, and inputting an auditing chain for each fragment in combination with a calling context and a semantic similar context of the fragment; and finally, summarizing all results to generate a report and cleaning temporary files. According to the method, full-amount deep review is realized, model limitation is broken through by combining code slicing and context, the code quality is comprehensively controlled, and the review efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of code review technology, and in particular to a method and apparatus for reviewing full code at the repository level. Background Technology

[0002] With the rapid development of the software development industry, code size and complexity are constantly increasing, making code quality control particularly important. In today's digital age, artificial intelligence (AI) technology is widely used in code review to improve efficiency and accuracy. However, existing AI-based code review methods have many problems that urgently need to be addressed.

[0003] On the one hand, most current AI-based code reviews only target the "differences" before and after modifications. In actual enterprise R&D processes, a comprehensive and in-depth evaluation of code quality is essential to ensure the standardization, security, and efficiency of the entire codebase, making full-scale code review a crucial step. For example, in some large projects, the addition of new features may potentially impact the existing, unmodified code logic. Reviewing only the modified parts may miss these potential issues, leading to compatibility problems, security vulnerabilities, and other issues.

[0004] On the other hand, forcibly inputting all the code into a large AI model for full review presents significant technical challenges. Although the context length that current large AI models can accept has improved, with some even exceeding 128K tokens, this is still insufficient to meet the needs of large codebases. Even if the code size does not exceed the context limit of the large model, it will still negatively impact the quality of the generated content. Research shows that models only achieve high content generation quality when the input code does not exceed 50% of its maximum context length. When the input code size approaches or exceeds this proportion, the model is prone to inaccurate content generation, logical inconsistencies, and other problems, making it unable to provide reliable code review results.

[0005] Furthermore, existing code review methods lack a deep understanding and utilization of code structure and semantics. Code is not merely a collection of text; it contains complex function call relationships, class inheritance structures, and more. Existing review methods based on simple text matching or local code analysis cannot grasp the overall architecture and logic of the code, making it difficult to discover deep-seated code quality issues.

[0006] In summary, existing AI-based code review technologies have significant shortcomings when facing the full-scale code review needs in actual enterprise R&D, both in terms of the completeness of the review scope and the effectiveness and accuracy in handling large-scale code input. There is an urgent need for a new and more effective repository-level full-scale code review method to solve these problems. Summary of the Invention

[0007] According to a first aspect of the present invention, a repository-level full code review method is provided, comprising the following steps: Download the target code repository from the code hosting platform to a local temporary directory; Scan the target code repository, identify and read the source code files within it; For source code files, based on abstract syntax tree parsing technology, each source code file is divided into multiple independent code segments according to predefined syntax structure types, and metadata containing file path, language type, structure type, symbol name and line number range is generated for each code segment; Code snippets are converted into vector representations using a pre-trained vector model and stored in a vector database to build a semantic index; Construct a review chain based on a large language model for multi-dimensional code review of individual code snippets; Construct a summary chain based on a large language model to aggregate the review results of multiple code snippets and generate a repository-level review report; Review all code snippets, including: A symbol index and a call relationship graph are constructed based on the metadata of code snippets; the symbol index is used to record the symbols defined in the code snippets; the call relationship graph is used to record the call relationships between code snippets. For the code snippet currently under review, perform the following operations: 1) Based on the call relationship graph, obtain other code segments called by the current code segment and use them as symbol definition contexts; 2) Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context; 3) Input the current fragment, the symbol definition context, and the semantically similar context into the review chain to obtain the review result of the current fragment; Input the review results of all code snippets into the summary chain to generate the final repository-level code review report; Delete the target code repository file in the local temporary directory.

[0008] Furthermore, the syntax structure types include function definitions, class definitions, and method definitions; for Python source code files, the ast module of the Python standard library is used for parsing and segmentation; for non-Python source code files, the tree-sitter library is used for parsing and segmentation.

[0009] Furthermore, constructing the call relationship graph includes: For each code snippet, the names of all external symbols called within its internal calls are extracted through analysis; A unique identifier is generated using the path and start and end line numbers of the code snippet as the caller key, and a mapping relationship is established using the list of called symbol names as the value.

[0010] Furthermore, the keywords of the current segment include at least one of the following: the name of the called symbol extracted from the call relationship graph, the symbol name of the current segment, the file name, the name of the imported module, and the dot call prefix.

[0011] Furthermore, multi-dimensional code review includes at least one of the following: code quality review, security vulnerability review, coding style review, comment integrity review, and sensitive information disclosure review.

[0012] According to a second aspect of the present invention, a repository-level full code review device is provided, comprising: The code retrieval module is used to download the target code repository from the code hosting platform to a local temporary directory. The code scanning module is used to scan the target code repository, identify and read the source code files within it; The code segmentation module is used to segment each source code file into multiple independent code fragments based on abstract syntax tree parsing technology and according to predefined syntax structure types, and generate metadata for each code fragment, including file path, language type, structure type, symbol name and line number range; The code conversion module is used to convert code snippets into vector representations through a pre-trained vector model and store them in a vector database to build a semantic index. The first building block is used to construct a review chain based on a large language model, which is used for multi-dimensional code review of individual code snippets; The second building module is used to build a summary chain based on a large language model, which is used to summarize the review results of multiple code snippets and generate a repository-level review report. The review module iterates through all code snippets for review, specifically including: A symbol index and a call relationship graph are constructed based on the metadata of code snippets; the symbol index is used to record the symbols defined in the code snippets; the call relationship graph is used to record the call relationships between code snippets. For the code snippet currently under review, perform the following operations: 1) Based on the call relationship graph, obtain other code segments called by the current code segment and use them as symbol definition contexts; 2) Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context; 3) Input the current fragment, the symbol definition context, and the semantically similar context into the review chain to obtain the review result of the current fragment; Input the review results of all code snippets into the summary chain to generate the final repository-level code review report; The cache deletion module is used to delete target code repository files in the local temporary directory.

[0013] Furthermore, the syntax structure types include function definitions, class definitions, and method definitions; for Python source code files, the ast module of the Python standard library is used for parsing and segmentation; for non-Python source code files, the tree-sitter library is used for parsing and segmentation.

[0014] Furthermore, constructing the call relationship graph includes: For each code snippet, the names of all external symbols called within its internal calls are extracted through analysis; A unique identifier is generated using the path and start and end line numbers of the code snippet as the caller key, and a mapping relationship is established using the list of called symbol names as the value.

[0015] Furthermore, the keywords of the current segment include at least one of the following: the name of the called symbol extracted from the call relationship graph, the symbol name of the current segment, the file name, the name of the imported module, and the dot call prefix.

[0016] Furthermore, multi-dimensional code review includes at least one of the following: code quality review, security vulnerability review, coding style review, comment integrity review, and sensitive information disclosure review.

[0017] A repository-level full code review method according to an embodiment of the present invention has the following beneficial effects: Comprehensive review: Breaking through the limitations of traditional methods that only review differences, this review performs a full audit of all code in the repository, effectively avoiding potential problems and comprehensively controlling code quality.

[0018] In-depth review: By constructing call graphs, vector indexes, and symbol definition indexes, we delve into the code structure and semantics to uncover deep-seated quality and security issues.

[0019] Adaptive Model: Transforms complete code into AST-based segments and related input segments, overcoming the contextual limitations of large models, improving review accuracy, and ensuring reliable results.

[0020] High-efficiency system: Based on the Large Language Model (LLM), the system constructs review and summary chains to examine individual fragments from multiple dimensions and generate reports, thereby improving review efficiency and saving manpower and time costs.

[0021] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0022] Figure 1 This is a flowchart of a repository-level full code review method according to an embodiment of the present invention.

[0023] Figure 2 This is a structural diagram of a warehouse-level full code review device according to an embodiment of the present invention. Detailed Implementation

[0024] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, further illustrating the present invention.

[0025] First, combine Figure 1 This invention describes a repository-level full code review method, which is used for code review and has a wide range of applications.

[0026] like Figure 1 As shown, a repository-level full code review method according to an embodiment of the present invention has the following steps: like Figure 1 As shown, in S1, the target code repository is downloaded from the code hosting platform to a local temporary directory. It should be noted that the code repository is downloaded to a temporary path as a zip archive using the APIs provided by GitHub or GitLab, combined with a personal or company-owned authentication key, and then unzipped. This step provides the foundational data for subsequent code scanning and processing.

[0027] like Figure 1 As shown, in S2, the target code repository is scanned, and its source code files are identified and read. It should be noted that for repositories hosted on GitHub and GitLab, the corresponding APIs or local file system access methods are used to perform a comprehensive scan of the files within the repository. During the scan, all source code files are accurately identified, laying the foundation for subsequent code processing. This step ensures that complete code data within the repository is obtained, which is the basis for full code review.

[0028] like Figure 1 As shown, in S3, for source code files, based on abstract syntax tree parsing technology, each source code file is divided into multiple independent code segments according to predefined syntax structure types, and metadata containing file path, language type, structure type, symbol name and line number range is generated for each code segment.

[0029] Furthermore, in this embodiment, the syntax structure types include function definitions, class definitions, and method definitions. For Python source code files, the `ast` module of the Python standard library is used for parsing and segmentation. For example, for a Python file containing multiple function and class definitions, the `ast` module can parse out the start and end positions of each function and class, thus achieving precise segmentation. Specifically, the code is first parsed using `ast.parse(code)` to generate a syntax tree. Then, a function `node_end_lineno` is defined to estimate the end line of a given `ast` node (compatible with cases where `end_lineno` is not available). By traversing the body nodes of the syntax tree, nodes of types such as `FunctionDef`, `AsyncFunctionDef`, and `ClassDef` are identified. Based on the start and end line numbers of the nodes, the code file is segmented into different Documents based on top-level functions, top-level classes, and the remaining code blocks at the top level. The metadata of each Document includes information such as path, language, structure type, name, and line number range.

[0030] For source code files in non-Python languages, such as Java files, the tree-sitter library is used for parsing and segmentation. The tree-sitter library supports syntax parsing for multiple languages. By loading the syntax rules of the Java language, it parses the Java file to generate an Abstract Syntax Tree (AST). Similarly, based on structures such as functions, classes, and methods, the Java file is segmented into multiple code fragments. Specifically, it first obtains the Java language parser and sets the language to Java. Then, it parses the code using `parser.parse(bytes(code, 'utf8'))` to generate a syntax tree. A depth-first traversal of the syntax tree is used to find nodes of target types such as `class_declaration`, `interface_declaration`, `method_declaration`, and `constructor_declaration`. The corresponding code fragments are extracted based on the start and end line numbers of the nodes, and their names (class / method names) are roughly extracted. A Document containing metadata such as path, language, type, name, and line number range is constructed. If no target type node is found, the entire file is treated as a single code fragment.

[0031] like Figure 1As shown, in S4, code snippets are converted into vector representations using pre-trained vector models (such as BAAI / bge-m3, microsoft / codebert-base, etc.) and stored in a vector database to build a semantic index. Vector representations can capture the semantic information of code snippets. When needed, by inputting keywords and utilizing FAISS's search function, semantically similar code snippets can be retrieved, providing semantically relevant context for subsequent review.

[0032] like Figure 1 As shown, in S5, a review chain based on a large language model is constructed for multi-dimensional code review of individual code snippets. It's important to note the selection of a suitable large language model (LLM), such as the GPT series. The review content is explicitly specified within the review chain. For example, for code quality review, rules are set to determine whether the code complexity is reasonable and whether there is redundant code; for security vulnerability review, checks are made for common security risks such as SQL injection and cross-site scripting attacks; for coding style review, based on the coding style followed by the project (such as PEP8 for Python, Google Java Style for Java), the code's indentation and naming conventions are checked; for comment integrity review, it is determined whether key logic parts in the code have corresponding comments; for sensitive information leakage review, it is detected whether the code contains sensitive information such as passwords and keys. The code snippet and its context are input into the large language model, and the review results for that code snippet are obtained based on the set review content.

[0033] like Figure 1 As shown, in S6, a summary chain based on the Large Language Model (LLM) is constructed to aggregate the review results of multiple code snippets and generate a repository-level review report. It should be noted that the summary chain is also built based on the selected Large Language Model (LLM). The review results of all code snippets are aggregated, analyzed, and summarized. For example, the frequency of various issues is counted, and different types of issues are prioritized. Finally, based on the results of the aggregated analysis, a full repository review report is generated. The report includes an overall code quality assessment, detailed descriptions and distribution of various issues, providing developers with a comprehensive and intuitive conclusion regarding the code repository review.

[0034] like Figure 1 As shown, in S7, all code snippets are reviewed, specifically including: S701: Construct symbol indexes and call relationship graphs based on metadata of code snippets; symbol indexes are used to record symbols defined in code snippets; call relationship graphs are used to record call relationships between code snippets.

[0035] It should be noted that: Symbol Index Construction: Construct a simple symbol index that only indexes documents with the kind "function", "class", or "method". By iterating through all documents and using "{language}::{symbol_name}" as the key, store the corresponding document in a list of values.

[0036] Call graph construction: For each Document, based on its language type, extract the function / method symbol names called in the code using ast or tree-sitter. For Python code, parse the code using the ast module, define the CallVisitor class to access Call nodes, and collect call symbols of type Name(id) and Attribute.attr. For Java code, parse the code using tree-sitter, traversing the syntax tree nodes to extract identifier child nodes as method / class names from method_invocation and object_creation_expression type nodes. Construct the call graph using caller_id="path:start_line - end_line" as the key and the list of called symbol names as the value.

[0037] S702: For the currently reviewed code snippet, perform the following operations: 1) Based on the call graph, obtain other code segments called by the current code segment as symbol definition contexts. Specifically, generate a caller_id based on the path, start_line, and end_line of the current code segment, obtain the set of called symbol names from the call graph, and then find the corresponding definition segment by symbol index.

[0038] 2) Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context. Specifically, extract keywords from the current code segment, including the called symbol name in the call graph, its own name, file name, imported module name, and dot prefix, and use these keywords to retrieve relevant code segments in the vector database.

[0039] 3) Input the current fragment, the symbol definition context, and the semantically similar context into the review chain to obtain the review result of the current fragment; S703: Input the review results of all code snippets into the summary chain to generate the final repository-level code review report; like Figure 1As shown, in S8, the target code repository files in the local temporary directory are deleted. Specifically, this involves deleting the zip archive downloaded to the temporary path and the folder extracted from it, in preparation for the next code review.

[0040] As described above, the repository-level full code review method according to an embodiment of the present invention has the following beneficial effects: Comprehensive review: Breaking through the limitations of traditional methods that only review differences, this review performs a full audit of all code in the repository, effectively avoiding potential problems and comprehensively controlling code quality.

[0041] In-depth review: By constructing call graphs, vector indexes, and symbol definition indexes, we delve into the code structure and semantics to uncover deep-seated quality and security issues.

[0042] Adaptive Model: Transforms complete code into AST-based segments and related input segments, overcoming the contextual limitations of large models, improving review accuracy, and ensuring reliable results.

[0043] High-efficiency system: Based on the Large Language Model (LLM), the system constructs review and summary chains to examine individual fragments from multiple dimensions and generate reports, thereby improving review efficiency and saving manpower and time costs.

[0044] The above combined with the appendix Figure 1 A repository-level full code review method according to an embodiment of the present invention is described. Furthermore, the present invention can also be applied to a repository-level full code review device.

[0045] like Figure 2 As shown, according to a second aspect of the present invention, a repository-level full code review device is provided, comprising: The code acquisition module 100 is used to download the target code repository from the code hosting platform to a local temporary directory; The code scanning module 200 is used to scan the target code repository, identify and read the source code files within it; The code segmentation module 300 is used to segment each source code file into multiple independent code segments based on abstract syntax tree parsing technology and according to predefined syntax structure types, and generate metadata for each code segment that includes file path, language type, structure type, symbol name and line number range; The code conversion module 400 is used to convert code snippets into vector representations through a pre-trained vector model and store them in a vector database to build a semantic index. The first building module 500 is used to build a review chain based on a large language model, which is used to conduct multi-dimensional code review of a single code snippet; The second building module 600 is used to build a summary chain based on a large language model, which is used to summarize the review results of multiple code snippets and generate a repository-level review report. The review module 700 is used to review all code snippets, specifically including: A symbol index and a call relationship graph are constructed based on the metadata of code snippets; the symbol index is used to record the symbols defined in the code snippets; the call relationship graph is used to record the call relationships between code snippets. For the code snippet currently under review, perform the following operations: 1) Based on the call relationship graph, obtain other code segments called by the current code segment and use them as symbol definition contexts; 2) Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context; 3) Input the current fragment, the symbol definition context, and the semantically similar context into the review chain to obtain the review result of the current fragment; Input the review results of all code snippets into the summary chain to generate the final repository-level code review report; The cache deletion module 800 is used to delete target code repository files in the local temporary directory.

[0046] Furthermore, the syntax structure types include function definitions, class definitions, and method definitions; for Python source code files, the ast module of the Python standard library is used for parsing and segmentation; for non-Python source code files, the tree-sitter library is used for parsing and segmentation.

[0047] Furthermore, constructing the call relationship graph includes: For each code snippet, the names of all external symbols called within its internal calls are extracted through analysis; A unique identifier is generated using the path and start and end line numbers of the code snippet as the caller key, and a mapping relationship is established using the list of called symbol names as the value.

[0048] Furthermore, the keywords of the current segment include at least one of the following: the name of the called symbol extracted from the call relationship graph, the symbol name of the current segment, the file name, the name of the imported module, and the dot call prefix.

[0049] Furthermore, multi-dimensional code review includes at least one of the following: code quality review, security vulnerability review, coding style review, comment integrity review, and sensitive information disclosure review.

[0050] As described above, the repository-level full code review device according to an embodiment of the present invention is an illustration of the device embodiment corresponding to the repository-level full code review method of the present invention. The relevant details are consistent with the embodiments of the invention, so they will not be repeated here.

[0051] It should be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0052] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A repository-level full code review method, characterized in that, It includes the following steps: Download the target code repository from the code hosting platform to a local temporary directory; Scan the target code repository, identify and read the source code files within it; For the source code file, based on abstract syntax tree parsing technology, each source code file is divided into multiple independent code segments according to the predefined syntax structure type, and metadata containing file path, language type, structure type, symbol name and line number range is generated for each code segment; The code snippets are converted into vector representations using a pre-trained vector model and stored in a vector database to build a semantic index. Construct a review chain based on a large language model for multi-dimensional code review of individual code snippets; Construct a summary chain based on a large language model to aggregate the review results of multiple code snippets and generate a repository-level review report; Review all code snippets, including: A symbol index and a call relationship graph are constructed based on the metadata of the code snippet; the symbol index is used to record the symbols defined in the code snippet; the call relationship graph is used to record the call relationships between code snippets; For the code snippet currently under review, perform the following operations: Based on the call relationship graph, other code segments called by the current code segment are obtained as symbol definition contexts; Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context; 3) Input the current segment, the symbol definition context, and the semantic similarity context into the review chain to obtain the review result of the current segment; Input the review results of all code snippets into the aggregation chain to generate the final repository-level code review report; Delete the target code repository file in the local temporary directory.

2. The repository-level full code review method as described in claim 1, characterized in that, The syntax structure types include function definitions, class definitions, and method definitions; for Python source code files, the ast module of the Python standard library is used for parsing and segmentation; for non-Python source code files, the tree-sitter library is used for parsing and segmentation.

3. The repository-level full code review method as described in claim 1, characterized in that, The construction call relationship graph includes: For each code snippet, the names of all external symbols called within its internal calls are extracted through analysis; A unique identifier is generated using the path and start and end line numbers of the code snippet as the caller key, and a mapping relationship is established using the list of called symbol names as the value.

4. The repository-level full code review method as described in claim 1, characterized in that, The keywords of the current segment include at least one of the following: the name of the called symbol extracted from the call relationship graph, the symbol name of the current segment, the file name, the name of the imported module, and the dot call prefix.

5. The repository-level full code review method as described in claim 1, characterized in that, The multi-dimensional code review includes at least one of the following: code quality review, security vulnerability review, coding standard review, comment integrity review, and sensitive information leakage review.

6. A warehouse-level full code auditing device, characterized in that, Include: The code retrieval module is used to download the target code repository from the code hosting platform to a local temporary directory. The code scanning module is used to scan the target code repository, identify and read the source code files within it; The code segmentation module is used to segment each source code file into multiple independent code segments based on abstract syntax tree parsing technology and according to predefined syntax structure types, and generate metadata for each code segment that includes file path, language type, structure type, symbol name and line number range; The code conversion module is used to convert the code fragments into vector representations through a pre-trained vector model and store them in a vector database to build a semantic index; The first building block is used to construct a review chain based on a large language model, which is used for multi-dimensional code review of individual code snippets; The second building module is used to build a summary chain based on a large language model, which is used to summarize the review results of multiple code snippets and generate a repository-level review report. The review module iterates through all code snippets for review, specifically including: A symbol index and a call relationship graph are constructed based on the metadata of the code snippet; the symbol index is used to record the symbols defined in the code snippet; the call relationship graph is used to record the call relationships between code snippets; For the code snippet currently under review, perform the following operations: Based on the call relationship graph, other code segments called by the current code segment are obtained as symbol definition contexts; Based on the keywords of the current segment, retrieve semantically similar code segments from the vector database as semantic similarity context; 3) Input the current segment, the symbol definition context, and the semantic similarity context into the review chain to obtain the review result of the current segment; Input the review results of all code snippets into the aggregation chain to generate the final repository-level code review report; The cache deletion module is used to delete target code repository files in the local temporary directory.

7. The warehouse-level full code review device as described in claim 6, characterized in that, The syntax structure types include function definitions, class definitions, and method definitions; for Python source code files, the ast module of the Python standard library is used for parsing and segmentation; for non-Python source code files, the tree-sitter library is used for parsing and segmentation.

8. The warehouse-level full code review device as described in claim 6, characterized in that, The construction call relationship graph includes: For each code snippet, the names of all external symbols called within its internal calls are extracted through analysis; A unique identifier is generated using the path and start and end line numbers of the code snippet as the caller key, and a mapping relationship is established using the list of called symbol names as the value.

9. The repository-level full code review device as described in claim 6, characterized in that, The keywords of the current segment include at least one of the following: the name of the called symbol extracted from the call relationship graph, the symbol name of the current segment, the file name, the name of the imported module, and the dot call prefix.

10. The repository-level full code review device as described in claim 6, characterized in that, The multi-dimensional code review includes at least one of the following: code quality review, security vulnerability review, coding standard review, comment integrity review, and sensitive information leakage review.

Images