A code structure view parsing and previewing method

Abstract

The present invention provides a code structure view parsing and previewing method, comprising a code preprocessing module, a code parsing module, a result processing module and a result storage module; the code preprocessing module is used to receive a code file transmitted by a requester, determine the language type of the code file, calculate the md5 of the code file content, and judge whether there is a historical analysis result; the code parsing module is used to analyze the code file, and parse it in combination with the language type involved in the code file, so as to obtain a grammar parse tree; and based on the visitor mode of the grammar parse tree, parse from the code file grammar entry rule to obtain the code structure analysis result; the present invention can realize the support of online preview of code structure views of almost all mainstream language files, and support target code jump and positioning, significantly improve code reading efficiency, and enhance ease of use.

Description

Technical Field

[0001] The present invention relates to the field of code analysis, and in particular to a code structure view parsing and previewing method. Background Art

[0002] The code structure view is when developers develop code in the IDE. The IDE will simultaneously provide the overall composition structure of the current code: for example, which class declarations, methods, function declarations, signature information such as corresponding parameters and return values, global declaration statements, etc. are included, so that developers can have a global preview of the overall structure of the current code, which is convenient for development work.

[0003] The technology used in the code structure view in the IDE is generally based on the code abstract syntax tree (AST) of a specific language for parsing. The abstract syntax tree is a tree structure obtained based on text code conversion. Each node corresponds to specific code, including line information, statement type, context, and child nodes. The code is generally converted from text to a computer executable program through the process of lexical analysis and syntax analysis, and then the abstract syntax tree is obtained, and then it is compiled and executed.

[0004] Code structure view Figure 1 It is usually integrated and displayed in the developer's local IDE (development tool), but in the common SaaS code hosting platform on the market, its code browsing page does not have a code overall structure view display function, which is not convenient for real-time preview of code structure and target code location. In addition, the existing technology is based on the abstract syntax tree: the code text needs to be converted into a tree structure for storage and traversal, which is relatively complicated and has relatively low parsing efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a code structure view parsing and previewing method to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A code structure view parsing and previewing method, comprising a code preprocessing module, a code parsing module, a result processing module and a result storage module;

[0008] The code preprocessing module is used to receive the code file transmitted by the requester, determine the language type of the code file, calculate the md5 of the code file content, and determine whether there is a historical analysis result;

[0009] The code parsing module is used to analyze the code file, and parse it in combination with the language type involved in the code file to obtain a syntax parse tree; and based on the visitor pattern of the syntax parse tree, parse from the code file syntax entry rule to obtain the code structure analysis result;

[0010] The result processing module is used to parse the code structure analysis results, describe them in a structured manner, and store the structured results persistently, and return the code structure analysis results to the requester for display; the structured results are stored according to the md5 of the code content as the key value, and if the md5 of the requested code content already exists in the historical analysis results, there is no need to repeat the analysis;

[0011] The result storage module is used to store structured results for easy query;

[0012] The specific steps are as follows:

[0013] Step 1: First, on the code browsing page, the requester initiates a code structure request to parse the code file and passes the code file to the code parsing service;

[0014] The code preprocessing module receives the code file and code file related parameters sent by the requester, and preprocesses the received content: analyzes the language type of the code file, calculates the md5 of the code file, calculates the size of the code file, and analyzes the code content of the code file;

[0015] Use md5 to determine whether the current code file has a historical analysis record. If it has been analyzed, you can directly query the historical analysis results and return them; if it has not been analyzed, proceed to the next step;

[0016] Step 2: Based on the code content and language type obtained in step 1, call the lexical and grammatical analysis program generated by the language based on Antlr4 to perform lexical analysis and grammatical analysis to obtain a grammatical analysis tree; and analyze the code structure based on the visitor pattern from the entry rules of the language grammar description;

[0017] Only rules related to the code file structure are analyzed, and sub-rules that are not of interest are not analyzed; when analyzing function declaration statements, only the function name and function signature are analyzed;

[0018] The structure of the code can be obtained by analyzing the top-level node and its child nodes in a targeted manner through the visitor pattern; if it is a function declaration statement, the function name, function parameter return value, and the file path and line number of the function can be further obtained based on the context information of the syntax parse tree node;

[0019] When a code file contains a syntax error, the grammar parse tree can still be generated according to the characteristics of Antlr4, ensuring that the analysis program can continue to analyze normally, thereby obtaining the code structure of the code file;

[0020] During the analysis process, the result processing module collects the code structure analysis results. The collected information includes the class declaration statements and related information, structure declarations and related information, function and method declaration statements and related information of the current analysis file;

[0021] Step 3: The structured result obtained by the analysis in step 2 is stored with the file md5 as the key value, and the code structure analysis result is returned to the requesting party;

[0022] Step 4: The requester receives the code structure analysis result, and the code platform displays the code structure view, thereby realizing online real-time preview of the code structure view, and performing code jump and positioning based on the file and line number of the statement.

[0023] As a further solution of the present invention: the code parsing module includes a lexical and grammatical analysis program and a grammatical parsing tree traversal program for generating commonly used development languages ​​based on Antlr4 and grammar description files of various commonly used development languages.

[0024] As a further solution of the present invention: the information included in the code structure analysis result includes: file package declaration, module declaration, dependent module import declaration, class definition, structure definition, function definition, function parameter list, global variable declaration, and constant declaration.

[0025] As a further solution of the present invention: the information included in the structured result includes: the code content, the line number, the column number, and the lexical type of each statement in the code file structure.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] After the present invention adopts the above method, it can support online preview of the code structure view of almost all mainstream language files, and support target code jump and positioning, significantly improving code reading efficiency and usability. When a user browses a certain language code file on the code platform, it can perform real-time parsing based on the present invention to obtain the overall code structure view of the current file for display. The user can clearly know which classes, methods, functions, global variable declarations and other overview information the current file contains. Click on a structure or method to locate the code line corresponding to the object in real time for browsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flowchart of a code structure view parsing and previewing method. DETAILED DESCRIPTION

[0029] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0030] See also Figure 1 A code structure view parsing and previewing method includes a code preprocessing module, a code parsing module, a result processing module and a result storage module; the code preprocessing module is used to receive the code file or code text content transmitted by the requester, determine the code file language type, calculate the code file content md5, and determine whether there is a historical analysis result. The code parsing module is used to analyze the code file, and parse it in combination with the language type involved in the code file to obtain a grammar parse tree; and based on the visitor pattern of the grammar parse tree, parse from the code file grammar entry rule to obtain the code structure analysis result.

[0031] The code parsing module includes syntax description files (.G4 suffix) based on Antlr4 and various commonly used development languages, and generates lexical and syntax analysis programs and syntax parsing tree traversal programs for commonly used development languages. The code structure analysis results generally include the following information (different languages ​​may have slightly different syntax): file package declarations, module declarations, dependent module import declarations, class definitions, structure definitions, function definitions, function parameter lists, global variable declarations, constant declarations and other statements.

[0032] The result processing module is used to parse the code structure analysis results, describe them in a structured way, and store the structured results persistently, and return the code structure analysis results to the requester for display; the structured results are stored according to the md5 of the code content as the key value, and if the md5 of the requested code content already exists in the historical analysis results, there is no need to repeat the analysis; the structured results contain information such as the code content, line number, column number, lexical type, etc. of each statement in the code file structure. The result storage module is used to store the structured results for easy query.

[0033] The specific steps are as follows:

[0034] Step 1: First, on the code browsing page, the requester (code host) initiates a code structure request to parse the code file, and passes the code file or code text content to the code parsing service;

[0035] The code preprocessing module receives the code file or code text content sent by the requester, as well as code file related parameters (file name, language, etc.), and preprocesses the received content: analyzes the language type of the code file, calculates the md5 of the code file, calculates the size of the code file, and analyzes the code content of the code file;

[0036] Use md5 to determine whether the current code file has a historical analysis record. If it has been analyzed, you can directly query the historical analysis results and return (the historical analysis results are saved in step 3); if it has not been analyzed, proceed to the next step;

[0037] Step 2: Based on the code content and language type obtained in step 1, call the lexical and grammatical analysis program generated by the language based on Antlr4 to perform lexical analysis and grammatical analysis to obtain a grammatical analysis tree; and analyze the code structure based on the visitor pattern from the entry rules of the language grammar description;

[0038] Only the rules related to the code file structure are analyzed, and the sub-rules that are not concerned are not analyzed. When analyzing the function declaration statement, only the function name and function signature and other information are analyzed. The code structure does not pay attention to the details inside the function body, and the function body can be ignored, that is, the function body rules and their sub-rules do not need to be accessed.

[0039] For example, the entry rule of the syntax parsing tree of the Go language is sourceFile, which represents the top-level node of a file syntax parsing tree. It contains sub-rules such as packageClause, importDecl, functionDecl|methodDecl|declaration, which correspond to package declaration, module import, function declaration, method declaration, and other declaration statements, and these declaration statements represent the overall structure of the file. Other languages ​​use similar analysis methods, starting with the entry rule of the language syntax file, and only analyzing the entry rule and sub-rules.

[0040] By using the visitor pattern to analyze the top-level node and its child nodes in a targeted manner, the structure of the code can be obtained, that is, which function declaration statements, method declaration statements, and other global declaration statements are contained in the code file; if it is a function declaration statement, the signature information such as the function name and function parameter return value, as well as the file path and line number where the function is located can be further obtained based on the context information of the syntax parse tree node; the corresponding information can be obtained similarly for method declarations and other declaration statements;

[0041] When a code file contains a syntax error, the grammar parse tree can still be generated according to the characteristics of Antlr4, ensuring that the analysis program can continue to analyze normally, thereby obtaining the code structure of the code file;

[0042] During the analysis process, the result processing module collects the code structure analysis results. The collected information includes the class declaration statements and related information, structure declarations and related information, function and method declaration statements and related information, and other global declaration statements and related information of the current analysis file.

[0043] Step 3: The code structure information obtained by the analysis in step 2 is stored with the file md5 as the key value, and the code structure analysis result is returned to the requesting party;

[0044] Step 4: The requester receives the code structure analysis result, and the code platform displays the code structure view, thereby realizing online real-time preview of the code structure view, and performing code jump and positioning based on the file, line number and other related information of the statement.

[0045] The working principle of the present invention is as follows: the present invention will perform code syntax analysis based on Antlr4 (an open source syntax parsing tool that can be used to build new languages, development tools or frameworks) and obtain a syntax parsing tree, parse the code structure based on the syntax parsing tree, and then generate a code structure preview view;

[0046] In terms of usage scenarios, it can be used in any code preview scenario other than IDE, including but not limited to online display of code structure on code hosting SaaS platform. It can also support target code structure positioning and jumping, making it convenient for developers to preview code.

[0047] In terms of technical implementation, this method has the following improvements compared to the analysis method based on the abstract syntax tree:

[0048] 1. Code structure analysis can be completed without relying on the abstract syntax tree of a certain language or traversing the entire syntax tree, thus improving analysis efficiency.

[0049] Based on the Antlr4 tool and the grammar description files of various languages, code lexical analysis, grammar analysis code programs and access programs for generating grammar analysis trees can be generated. The generated code programs can parse the target code to generate grammar analysis trees CST (grammar analysis trees are structures with context information obtained after grammar analysis according to the rules in the grammar description files of Anlr4 (.G4 suffix))

[0050] Antlr4 provides two parse tree processing modes: monitoring mode and visitor mode. This method will use the visitor mode.

[0051] The monitoring mode will automatically traverse the entire parse tree, and notify the listener when traversing each tree node, thereby completing the analysis action set by the listener. This process is similar to the traversal analysis of the entire abstract syntax tree (AST), including each child node. Because we are only concerned about the overall structure of the file, we do not pay attention to the details of the function or method, and do not need to analyze it in depth, so the monitoring mode will do a lot of useless traversal for code structure analysis. Based on the visitor mode, it is not necessary to traverse all child nodes, and the child nodes that need to be analyzed by the analysis program can be independently controlled, and only specific types of nodes are analyzed.

[0052] 2. Files containing grammatical errors can be analyzed normally. Based on the self-repair feature of Antlr4 grammar parsing process, grammar parsing can be continued. However, based on the abstract syntax tree, the abstract syntax tree cannot be correctly generated due to grammatical errors, and thus analysis cannot continue.

[0053] A conventional abstract syntax tree generation module cannot correctly generate an abstract syntax tree once a syntax error is encountered when parsing a code file, and therefore cannot complete subsequent code analysis based on the abstract syntax tree.

[0054] During the lexical parsing process, the Antlr4 tool will use the pre-detection technology to match the next expected lexical symbol for places that do not conform to the grammatical specifications, and then perform subsequent lexical parsing and grammatical parsing to obtain a grammatical parsing tree to meet the needs of subsequent code analysis.

[0055] 3. Almost all languages ​​can be analyzed based on a set of frameworks. If it is based on the abstract syntax tree technology, it is necessary to use the corresponding language technology stack to obtain the abstract syntax tree structure for analysis according to the specific language, which has a high development cost.

[0056] Currently, the Antlr4 open source tool already supports grammar rule description files of almost all common languages. Based on these description files, the lexical and grammar parsing files of a specific language can be quickly generated through the Antlr4 command, and runtime libraries for multiple development languages ​​are provided, so that developers can implement code analysis programs for other languages ​​based on a certain language. (For example, based on the Go language, combined with the Go language runtime of antlr4, analysis programs for C++, Java, PHP, Go, Python and other codes can be implemented. In this way, analysis programs can be developed based on the same framework, and there is no need to switch technology stacks according to different languages, thereby reducing development costs. If based on abstract syntax tree analysis, when analyzing Go language code, it is necessary to generate an abstract syntax tree based on the Go language AST module, and develop an analysis program based on the Go language; when analyzing Python language code, it is necessary to switch to Python language for development, and a new technology stack must be used for each language analyzed.)

[0057] The above describes in detail the preferred implementation of this patent, but this patent is not limited to the above implementation. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of this patent.

Claims

1. A code structure view parsing and previewing method, characterized in that: It includes a code preprocessing module, a code parsing module, a result processing module and a result storage module; The code preprocessing module is used to receive the code file transmitted by the requester, determine the language type of the code file, calculate the md5 of the code file content, and determine whether there is a historical analysis result; The code parsing module is used to analyze the code file and parse it in combination with the language type involved in the code file to obtain a syntax parsing tree; Based on the visitor pattern of the syntax parse tree, the code file is parsed from the syntax entry rules to obtain the code structure analysis results; The result processing module is used to parse the code structure analysis results, describe them in a structured manner, store the structured results persistently, and return the code structure analysis results to the requester for display; The structured results are stored using the md5 of the code content as the key value. If the md5 of the requested code content already exists in the historical analysis results, there is no need to repeat the analysis; The result storage module is used to store structured results for easy query; The specific steps are as follows: Step 1: First, on the code browsing page, the requester initiates a code structure request to parse the code file and passes the code file to the code parsing service; The code preprocessing module receives the code file and code file related parameters sent by the requester, and preprocesses the received content: analyzes the language type of the code file, calculates the md5 of the code file, calculates the size of the code file, and analyzes the code content of the code file; Use md5 to determine whether the current code file has a historical analysis record. If it has been analyzed, directly query the historical analysis results and return them; If it has not been analyzed, proceed to the next step; Step 2: Based on the code content and language type obtained in step 1, call the lexical and grammatical analysis program generated by the language based on Antlr4 to perform lexical analysis and grammatical analysis to obtain a grammatical analysis tree; And analyze the code structure based on the visitor pattern from the entry rules described in the language's syntax; Only rules related to the code file structure are analyzed, and sub-rules that are not of interest are not analyzed; when analyzing function declaration statements, only the function name and function signature are analyzed; The structure of the code can be obtained by analyzing the top-level node and its child nodes in a targeted manner through the visitor pattern; If it is a function declaration statement, further obtain the function name and function parameter return value, as well as the file path and line number where the function is located based on the context information of the syntax parse tree node; When a code file contains a syntax error, a syntax parse tree is still generated according to the characteristics of Antlr4 to ensure that the analysis program continues to analyze normally, thereby obtaining the code structure of the code file; During the analysis process, the result processing module collects the code structure analysis results. The collected information includes the class declaration statements and related information, structure declarations and related information, function and method declaration statements and related information of the current analysis file; Step 3: The structured result obtained by the analysis in step 2 is stored with the file md5 as the key value, and the code structure analysis result is returned to the requesting party; Step 4: The requester receives the code structure analysis result, and the code platform displays the code structure view, thereby realizing online real-time preview of the code structure view, and performing code jump and positioning based on the file and line number of the statement.

2. A code structure view parsing and previewing method according to claim 1, characterized in that: The code parsing module includes a lexical and grammatical analysis program and a grammar parsing tree traversal program for generating commonly used development languages ​​based on Antlr4 and grammar description files of various commonly used development languages.

3. A code structure view parsing and previewing method according to claim 1, characterized in that: The code structure analysis result includes the following information: package declaration, module declaration, dependent module import declaration, class definition, structure definition, function definition, function parameter list, global variable declaration, and constant declaration of the file.

4. A code structure view parsing and previewing method according to claim 1, characterized in that: The structured result includes the following information: code content, line number, column number, and lexical type of each statement in the code file structure.

Images