AI native code analysis engine implementation method and device

Abstract

The invention discloses an AI native code analysis engine implementation method and device, and the device comprises an AI native code analysis engine unit and a large language model. The AI native code analysis engine unit comprises a code preprocessing module, a code analysis rule management module, a large model cue word management module, a code analysis engine module, an analysis result processing module and a scanning result storage module. The capability that one scanning engine supports code analysis of almost all language files and different dimensions is achieved, rule implementation logic is based on natural language organization, the development cost of a code analysis tool is greatly reduced, analysis dimensions and analysis rules can be conveniently expanded, and the analysis efficiency is improved. According to the method, the requirement of a terminal user for customizing the code scanning rule can be well met, and it is not needed to write custom rule codes like a traditional code analysis tool or autonomously achieve custom rules based on DSL grammar.

Description

Technical Field

[0001] The present invention relates to the field of code analysis technology, and specifically to a method and device for implementing an AI-native code analysis engine. Background Art

[0002] Traditional code analysis tools are based on lexical and syntactic analysis of code. They use code to implement code analysis rules and logic, analyze the code's grammatical structure, defects in the code, and measure the code.

[0003] The shortcomings of traditional code analysis tools are:

[0004] 1. The technical threshold for developing code analysis tools is relatively high. You need to understand the syntax of a certain language, understand lexical analysis, syntax analysis, and the concept of abstract syntax trees, and then develop corresponding code in combination with abstract syntax trees;

[0005] Abstract syntax tree refers to the process of converting code from text to computer executable program, which generally goes through lexical analysis and grammatical analysis, and then obtains the abstract syntax tree, and then performs subsequent compilation and execution. Figure 4 shown.

[0006] 2. User-defined requirements are not flexible enough. For example, if a user wants to customize a new scanning rule based on a code defect scanning tool, the tool maintainer needs to develop the scanning rule and implement the rule detection logic code to meet the user's customization requirements.

[0007] 3. Traditional code analysis tools have a limited range of supported languages. Generally, a tool only has the ability to analyze a specific code language. For example, Cppcheck only analyzes C and C++ codes. To analyze codes in other languages, another tool must be used.

[0008] In order to solve the above problems, this application proposes an AI-native code analysis engine implementation method and device. Summary of the Invention

[0009] The purpose of the present invention is to provide an AI-native code analysis engine implementation method and device to solve the problems raised in the above background technology.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] A method and device for implementing an AI-native code analysis engine include an AI-native code analysis engine unit and a large language model. The AI-native code analysis engine unit includes a code preprocessing module, a code analysis rule management module, a large model prompt word management module, a code analysis engine module, an analysis result processing module, and a scanning result storage module. The large language model includes a private deployment model, a large model official API, or a third-party cloud platform API.

[0012] As a further solution of the present invention: the code preprocessing module is responsible for preprocessing the code directory or file or code text content to be analyzed, calculating the number of files to be analyzed, obtaining basic information such as the file suffix and the language to which the code belongs, and reading the code content of the file.

[0013] As a further solution of the present invention: the code analysis rule management module is used for analyzing different rules. The rules are the implementation logic of the analysis code, and are divided into different rule types according to the dimensions of the analysis code. The rule types include but are not limited to the following dimensions: code defect dimension, code measurement dimension, and code structure analysis dimension.

[0014] As a further solution of the present invention: the code defect dimension is a series of defect inspection rules according to the detection of different defect types, each defect inspection rule has basic constraint information and rule detection logic, the code measurement dimension includes but is not limited to the total number of statistical code lines, the number of executable code lines, the number of annotated code lines, the code annotation rate, the file circle complexity and other indicators, the code structure analysis dimension includes the grammatical structure of the parsed code, the classes contained and which member variables and methods each class contains, as well as the calling relationship between each method and other information, which structures and interfaces are contained.

[0015] As a further solution of the present invention: the large model prompt word management module will use different prompt words to interact with the large language model for different analysis dimensions, including the role definition of the large language model, the specific format requirements for the structured output of the analysis results, and the relevant auxiliary information provided to the large language model. The large model prompt word management module maintains different prompt word templates according to different analysis dimensions, and generates prompt words with complete rules based on the corresponding templates when performing specific rule analysis.

[0016] As a further solution of the present invention: the code analysis engine module is used to process each code file to be analyzed, execute analysis rules in combination with different analysis dimensions, that is, interact with the large model based on the assembled complete prompt words, receive the analysis results output by the large language model after the execution of each rule, and statistically analyze the overall progress.

[0017] As a further solution of the present invention: the analysis result processing module is responsible for collecting and parsing the structured results output by the received large model, and storing the data according to business needs. The data can be stored in any form such as database or file for the business party to use the data.

[0018] As a further solution of the present invention: the scanning result storage module is used to store the result data of each analysis rule and the overall summary scanning result and other data, and the storage is not limited to the form of database or disk.

[0019] As a further solution of the present invention: the method steps are as follows:

[0020] Step 1: Based on the input code directory or file to be scanned or the complete code content, calculate the number of files to be analyzed, obtain the file suffix, code language and other basic information, read the file code content as the object of subsequent code analysis;

[0021] Step 2: The code analysis engine module analyzes the acquired code content based on different dimensions and rules. The analysis dimensions include, but are not limited to, code defect analysis, code metrics, and code structure analysis. Each dimension contains N analysis rules. The analysis engine is responsible for executing these rules in sequence and calculating the overall analysis progress.

[0022] Step 3: For each rule in step 2, the code analysis engine module obtains the complete prompt word corresponding to each rule in turn based on the code analysis rule management module and the large model prompt word management module, and interacts with the large language model. Each request to the large language model executes a rule, and waits for the analysis output of the large language model;

[0023] Step 4: The analysis result processing module collects the structured results output by the large language model after each rule is executed, parses them according to business needs, and stores the analysis results. The storage format can be in-memory storage, database persistent storage, or other formats;

[0024] Step 5: After all rules are executed and the output results are parsed, the code analysis engine module summarizes the analysis results of each rule and generates the final result file.

[0025] As a further solution of the present invention: the result file in step five is a file in xml or json format.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention can leverage the code understanding capabilities of the AI large model to enable a scanning engine to support code analysis in almost all language files and different dimensions. The rule implementation logic is based on natural language organization, which greatly reduces the development cost of the code analysis tool and facilitates the expansion of analysis dimensions and analysis rules. It can well meet the needs of end users to customize code scanning rules without the need to write custom rule code or independently implement custom rules based on DSL syntax like traditional code analysis tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A block diagram of a method and apparatus for implementing an AI-native code analysis engine.

[0029] Figure 2 A defect detection rule diagram for a method and apparatus for implementing an AI-native code analysis engine.

[0030] Figure 3 This figure shows an example of the output effect of the model analysis results in the AI-native code analysis engine implementation method and device.

[0031] Figure 4 is the existing abstract syntax tree diagram. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] See also Figures 1 to 4 In an embodiment of the present invention, a method and device for implementing an AI native code analysis engine include an AI native code analysis engine unit and a large language model. The AI native code analysis engine unit includes a code preprocessing module, a code analysis rule management module, a large model prompt word management module, a code analysis engine module, an analysis result processing module, and a scanning result storage module. The large language model includes a private deployment model, a large model official API, or a third-party cloud platform API. The code preprocessing module is responsible for preprocessing the code directory or file or code text content to be analyzed, calculating the number of files to be analyzed, obtaining basic information such as the file suffix and the language to which the code belongs, and reading the code content of the file; the code analysis rule management module is used for analyzing different rules. The rules are the implementation logic of the analysis code, and are divided into different rule types according to the dimensions of the analysis code. The rule types include but are not limited to the following dimensions: code defect dimension, code measurement dimension, code structure analysis dimension, etc.

[0034] The code defect dimension has a series of defect inspection rules according to the detection of different defect types. Each defect inspection rule has basic constraint information and rule detection logic. For example, there is a rule to detect whether there is a defect of variable self-assignment statement in the code, and a unique identifier such as "VariableSelfAssignChecker" is defined for the rule. Another rule is to detect whether there is a defect with the same if-else branch logic in the code. The unique identifier of the rule is such as "BranchWithSameImplChecker". And so on. N defect rules can be defined. Each defect rule uses natural language to define the rule description information and the logic of the rule detection code. In this way, the code defect rule can support users to customize the rules through natural language, which greatly improves the scalability and ease of use.

[0035] Example of defect rule description:

[0036] VariableSelfAssignChecker

[0037] Scan rule constraints include:

[0038] Rule Name: Detect variable self-assignment statements;

[0039] Rule identifier Ident: VariableSelfAssignChecker;

[0040] Rule Level: MIDDLE, indicating a medium severity level.

[0041] The rule detection logic includes:

[0042] Focus on detecting variable assignment statements in the code;

[0043] If the variable name on the left side of the assignment statement is equal to the value on the right side, the rule detection logic is hit;

[0044] If the detection logic is hit, it indicates that the line of code has a code defect described by the rule.

[0045] The code metric dimensions include but are not limited to the total number of statistical code lines, the number of executable code lines, the number of commented code lines, the code comment rate, the file circle complexity and other indicators.

[0046] The rule description corresponding to code metrics is as follows: CodeMetricsRule

[0047] Measure code content with multiple indicators

[0048] Count the total number of code lines loc, the number of executable code lines nloc, and the number of comment code lines comment respectively;

[0049] Calculate the code comment rate, using the formula: ... (e.g., rate = comment / loc * 100%);

[0050] The cyclomatic complexity cycle_complexity of the statistical file and the calculation rules of cyclomatic complexity are described.

[0051] The code structure analysis dimension includes the grammatical structure of the parsed code, the classes contained and the member variables and methods contained in each class, the calling relationship between each method, and other information, including the structures and interfaces contained;

[0052] The corresponding rule description is as follows: CodeStructureRule;

[0053] Analyze the code structure;

[0054] If it is an object-oriented language and there is a concept of class in the grammar, all class names contained in the code are parsed;

[0055] Further analyze the member variables and member methods contained in each class;

[0056] Overall analysis of the calling relationship between each method;

[0057] And so on for other detection logic.

[0058] The large model prompt word management module will use different prompt words to interact with the large language model for different analysis dimensions, including the role definition of the large language model, the specific format requirements for the structured output of analysis results, and the relevant auxiliary information provided to the large language model. The large model prompt word management module maintains different prompt word templates according to different analysis dimensions, and generates complete prompt words based on the corresponding templates when performing specific rule analysis;

[0059] For the code defect dimension, the prompt will require the large model to perform a defect check on the code, define the field information contained in each output defect, and the output format in which the results are output, such as XML or JSON.

[0060] refer to Figure 2 A complete prompt word demo example based on a template for a defect inspection rule is shown below: "Scanning rule constraints" and "Rule detection logic" are specific rule implementations. These parts can be replaced with any other defect inspection rule. The rest of the prompt word can be used as a prompt word template for the defect inspection rule. The corresponding code content can be replaced during analysis.

[0061] The following is an example of the output effect of the corresponding large language model analysis results:

[0062] The large language model provides analysis results that meet the requirements according to the rule constraints and format output requirements;

[0063] For the code metric dimension, the prompt will require the large language model to analyze the code metrics, perform appropriate calculations as required, and output the metric data in the required format to facilitate subsequent result analysis;

[0064] For code structure analysis, the large language model is required to analyze the code from a holistic perspective, counting classes, variables, methods, and so on, including the number, specific names, lines, start and end line numbers, and output structured analysis results as required to facilitate subsequent result analysis.

[0065] The code analysis for other dimensions is the same as above, and there will also be corresponding prompt word templates;

[0066] The code analysis engine module is used to process each code file to be analyzed, execute analysis rules in combination with different analysis dimensions, that is, interact with the large model based on the assembled complete prompt words, receive the analysis results output by the large language model after the execution of each rule, and count the overall progress of the analysis.

[0067] Here is an example:

[0068] To analyze 10 code files in a certain directory, process each file in turn, perform code defect analysis, code measurement, and code structure analysis on each file, execute all defect rules, code measurement rules, and code structure analysis rules in turn. Each rule will be assembled into a complete prompt word according to the prompt word template of the corresponding dimension and then interact with the big model. The big model mentioned here can be a big model privately deployed within the enterprise or the API interface of the mainstream SaaS version big model to obtain the analysis results of each rule. After executing all rules of different dimensions of all files, the analysis engine will collect all analysis results.

[0069] The analysis result processing module is responsible for collecting and parsing the structured results output by the received large model, and storing the data according to business needs. The data can be stored in any form such as a database or file for use by the business party.

[0070] The scanning result storage module is used to store the result data of each analysis rule and the overall summary scanning result and other data, and the storage is not limited to the form of database or disk;

[0071] The implementation method of the AI native code analysis engine is as follows:

[0072] Step 1: Based on the input code directory or file to be scanned or the complete code content, calculate the number of files to be analyzed, obtain the file suffix, code language and other basic information, read the file code content as the object of subsequent code analysis;

[0073] Step 2: The code analysis engine module analyzes the acquired code content based on different dimensions and rules. The analysis dimensions include, but are not limited to, code defect analysis, code metrics, and code structure analysis. Each dimension contains N analysis rules. The analysis engine is responsible for executing these rules in sequence and calculating the overall analysis progress.

[0074] Step 3: For each rule in step 2, the code analysis engine module obtains the complete prompt word corresponding to each rule in turn based on the code analysis rule management module and the large model prompt word management module, and interacts with the large language model. Each request to the large language model executes a rule, and waits for the analysis output of the large language model;

[0075] Step 4: The analysis result processing module collects the structured results output by the large language model after each rule is executed, parses them according to business needs, and stores the analysis results. The storage format can be in-memory storage, database persistent storage, or other formats;

[0076] Step 5: After all rules are executed and the output results are parsed, the code analysis engine module summarizes the analysis results of each rule and generates a final result file in XML or JSON format.

[0077] Differences in the implementation and management of scanning rules:

[0078] The main differences between the present invention and the existing traditional code analysis technology are:

[0079] 1. Traditional code analysis is based on the abstract syntax tree to develop individual scanning rules. Each rule needs to implement specific code detection logic. The scanning tool needs to manage individual code files and execute rule code files to obtain code analysis results.

[0080] The present invention directly describes the detection logic of the rules based on natural language, does not require specific code implementation, and manages text-type files or markdown format files. The present invention is simpler and more flexible when expanding and customizing rules.

[0081] 2. Traditional scanning tools are based on abstract syntax trees and combine grammar rules to implement specific rule checking logic to analyze whether the code has defects, or count code indicators and analyze code structure based on syntax tree node types;

[0082] The basic principle of this invention is to perform code analysis based on the large model's ability to understand the code language, combined with rule detection logic description information and prompt words. It is an AI native code analysis engine tool.

[0083] The present invention can make use of the code understanding ability of the AI large model to realize the ability of a scanning engine to support code analysis of almost all language files and different dimensions. The implementation logic of the rules is based on natural language organization, which greatly reduces the development cost of the code analysis tool and facilitates the expansion of analysis dimensions and analysis rules. It can well meet the needs of end users to customize code scanning rules without the need to write custom rule code (such as sonarqube plug-ins) like traditional code analysis tools or to independently implement custom rules based on DSL syntax (such as github's CodeQL based on sql syntax to write rules, Huawei's CodeNavi to write dsl syntax rules).

[0084] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An AI-native code analysis engine implementation device, comprising an AI-native code analysis engine unit and a large language model, characterized in that: The AI native code analysis engine unit includes a code preprocessing module, a code analysis rule management module, a large model prompt word management module, a code analysis engine module, an analysis result processing module and a scanning result storage module. The large language model includes a private deployment model, a large model official API or a third-party cloud platform API.

2. The method and apparatus for implementing an AI-native code analysis engine according to claim 1, characterized in that: The code preprocessing module is responsible for preprocessing the code directory or file or code text content to be analyzed, calculating the number of files to be analyzed, obtaining the file suffix and the language to which the code belongs, and reading the code content of the file.

3. The AI-native code analysis engine implementation device according to claim 1, characterized in that: The code analysis rule management module is used for analyzing different rules. Rules are the implementation logic of the analysis code and are divided into different rule types according to the dimensions of the analysis code. The rule types include but are not limited to the following dimensions: code defect dimension, code measurement dimension, and code structure analysis dimension.

4. The AI-native code analysis engine implementation device according to claim 3, characterized in that: The code defect dimension is a series of defect inspection rules based on the detection of different defect types. Each defect inspection rule has basic constraint information and rule detection logic. The code measurement dimension includes but is not limited to the total number of statistical code lines, the number of executable code lines, the number of annotated code lines, the code annotation rate, and the file circle complexity. The code structure analysis dimension includes the grammatical structure of the parsed code, the classes included, the member variables and methods contained in each class, as well as the calling relationship information between each method, and the structures and interfaces contained.

5. The AI-native code analysis engine implementation device according to claim 1, characterized in that: The large model prompt word management module will use different prompt words to interact with the large language model for different analysis dimensions, including the role definition of the large language model, the specific format requirements for the structured output of the analysis results, and the relevant auxiliary information provided to the large language model. The large model prompt word management module maintains different prompt word templates according to different analysis dimensions, and generates prompt words with complete rules based on the corresponding templates when performing specific rule analysis.

6. The AI-native code analysis engine implementation device according to claim 1, characterized in that: The code analysis engine module is used to process each code file to be analyzed, execute analysis rules in combination with different analysis dimensions, that is, interact with the large model based on the assembled complete prompt words, receive the analysis results output by the large language model after the execution of each rule, and count the overall progress of the analysis.

7. The AI-native code analysis engine implementation device according to claim 1, characterized in that: The analysis result processing module is responsible for collecting and parsing the structured results output by the received large model, and storing the data according to business needs. The data can be stored in the form of a database or a file for the business party to use the data.

8. The AI-native code analysis engine implementation device according to claim 1, characterized in that: The scanning result storage module is used to store the result data of each analysis rule and the overall summary scanning result data, and the storage is not limited to the database or disk form.

9. The method for implementing an AI-native code analysis engine according to claim 1, wherein: The method steps are as follows: Step 1: Based on the input code directory or file or complete code content to be scanned, calculate the number of files to be analyzed, obtain the file suffix, the language to which the code belongs, and read the code content of the file as the object of subsequent code analysis; Step 2: The code analysis engine module analyzes the acquired code content based on different dimensions and rules. The analysis dimensions include, but are not limited to, code defect analysis, code metrics, and code structure analysis. Each dimension contains N analysis rules. The analysis engine is responsible for executing these rules in sequence and calculating the overall analysis progress. Step 3: For each rule in step 2, the code analysis engine module obtains the complete prompt word corresponding to each rule in turn based on the code analysis rule management module and the large model prompt word management module, and interacts with the large language model. Each request to the large language model executes a rule, and waits for the analysis output of the large language model; Step 4: The analysis result processing module collects the structured results output by the large language model after each rule is executed, parses them according to business needs, and stores the analysis results. The storage format can be in-memory storage, database persistent storage, or other formats; Step 5: After all rules are executed and the output results are parsed, the code analysis engine module summarizes the analysis results of each rule and generates the final result file.

10. The method for implementing an AI-native code analysis engine according to claim 9, characterized in that: The result file in step 5 is a file in XML or JSON format.

Images