A method based on antlr tool to support the compilation of super large source code files
By traversing the tokens of each line of statements in the antlr tool for encoding and storing them in temporary files, the problem of rapid memory growth when antlr compiles super-large source files is solved, and a stable compilation process is realized, avoiding system jamming and stuttering.
Patent Information
- Application Number
- CN202411421800.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-10-12
AI Technical Summary
When using the antlr tool to compile super-large source files, traditional methods lead to a sharp increase in memory, resulting in a stuttering or black screen of the operating system. This is mainly because antlr generates a large amount of temporary memory during accessing the token tree and the memory usage during lexical compilation is too large.
By introducing a method in the antlr tool, encode and store it in a temporary file when traversing the token of each line of statement, instead of keeping it in memory, and store the token memory set and context content in memory only when needed, reducing memory usage , and write temporary file contents to the target file at the end of compilation.
It effectively reduces the memory usage during the compilation process, avoids system unresponsiveness and crashes, ensures that the memory usage is stable when compiling super large files, and can complete the compilation smoothly.
Smart Images
Figure CN118939276B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of source code file compilation, in particular to a method for supporting the compilation of super large source code files based on an antlr tool. Background Art
[0002] antlr is a powerful parser generator, which supports the generation of lexical analyzers and syntax analyzers for custom programming languages. The main uses of antlr include implementing data loaders, language interpreters, language translators, etc. It parses files based on automatically generated syntax analysis trees, and is mainly aimed at the semiconductor testing field.
[0003] The Antlr tool generates a language lexical tree, a parsing tree, and a source code project for accessing the tree according to the grammatical rules of the language. The project is used to compile the language source code. Since the vistor will save the node in the memory when traversing the node in the tree. If the source code file is large, the memory will also increase synchronously. Finally, the node information in the memory is taken out and compiled to generate the file. In the traditional source code compilation process, three memory collections are generated. If the source code is large, the memory will increase sharply when there are many tokens in each line of source code. When compiling more than 5 million lines of source code, the memory will quickly increase to 10G, and the operating system will be stuck or even black screen. The essence of the problem is that antlr generates a large amount of temporary memory in the process of accessing the token tree, and also generates a large amount of memory when compiling the source code according to the lexical structure. For this reason, a method based on the antlr tool to support the compilation of super large source code files is provided to solve the above problems. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides a method for supporting the compilation of super-large source code files based on the antlr tool, which solves the problem of three memory collections generated in the traditional source code compilation process. If the source code is large and the number of tokens per line of source code is large, the memory will increase sharply. When compiling more than 5 million lines of source code, the memory will quickly increase to 10G, and the operating system will be stuck or even black screen.
[0005] To achieve the above objectives, the present invention is implemented by the following technical solutions: a method for supporting the compilation of super large source code files based on antlr tools, comprising the following steps:
[0006] S1. Source code parsing: start the antlr tool, read the source code file through the antlr runtime library, and perform source code parsing;
[0007] S2, token encoding, after the source code is parsed, it is stored in the antlr runtime library memory, and the visitor of the token tree generated by antlr is used to parse line by line, and the public type token is stored in the memory;
[0008] S3. Token judgment in the function: parse a large number of tokens in the function to determine their validity. If the current line statement is valid, encode the token in the current line and save it in a temporary file, and delete the token of the current statement from the memory. If the current line statement is invalid, the compilation ends, prompts an error, and returns.
[0009] S4. Judge the public part token in the memory. When the last line is parsed and transcoded, judge the public part token in the memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, prompts an error, and returns.
[0010] S5. Compile file output. The compilation result in the programcontext memory in antlr is directly written into the target compilation file. Then the temporary file is read and the code in the temporary file is added to the corresponding position of the target compilation file to complete the compilation of the super large source code file.
[0011] Preferably, the antlr optimizes the storage structure, and the final vector content only includes common part information such as header, vectorList, etc., and also includes token and label information used for jumping.
[0012] Preferably, in step S1, when parsing the source code, the common part tokens parsed out of the token tree are stored in the programcontext memory, and at the same time, in the token tree, all tokens for parsing function statements are stored in the normalvector temporary memory to form a token memory set, and this part of the tokens is directly deleted from the programcontext memory, the encoding memory is no longer retained, and finally all the tokens in the normalvector temporary memory are encoded, and the encoding results are stored in a temporary file, and are repeatedly executed when the visitor of the token tree generated by antlr traverses the tokens corresponding to each line of statements.
[0013] Preferably, after completing all source code parsing, the antlr directly writes the compilation results in the programcontext memory into the target compilation file when processing the data therein, and writes the encoding results in the temporary file into the corresponding offset position of the target compilation file.
[0014] Preferably, in steps S3 and S4, when judging the token in the function and the public part token in the memory, the visitor traverses the token tree to read each line of source code in the function, and directly judges according to the settings of the grammar rule g4 file and other specific rules defined by the language. The judgment is successful if it conforms to the regular expression grammar. If the judgment is successful, the code is stored in a temporary file, and the token corresponding to the line of source code visited by the vistor is deleted in the memory, and the code of other lines of statements traversed subsequently is appended to the temporary file.
[0015] Preferably, after the steps S3 and S4 are completed, the token types belonging to the public part that may be called by other tokens are stored in the memory until the syntax is checked again at the end, and they are transcoded and written to the corresponding position of the compiled target file.
[0016] Preferably, in step S5, the specific steps of compiling the file output are as follows:
[0017] Step 1: Encode the public part and write the result into the target file;
[0018] Step 2: Write the split code between the common part and the function statement;
[0019] Step 3: Read the temporary file content and write it into the target file in binary form;
[0020] Step 4: Write the remaining necessary parts into the target file;
[0021] Step 5: Form the final target file.
[0022] The present invention discloses a method for supporting the compilation of super large source code files based on antlr tool, which has the following beneficial effects:
[0023] 1. This method based on antlr tool supports the compilation of super large source code files. When vistor traverses the tokens corresponding to each line of statements, these tokens will be encoded and stored in temporary files and token memory sets. The encoded memory will no longer be retained. The content that needs to be compiled in connection with the context is stored in the memory. This part of the content accounts for a small number, such as jumps, calls to other functions, etc. When the parsing and compilation reaches the last statement, the temporary file is combined with other information to form a complete compiled file. The entire memory is mainly the memory size stored in the antlr parsing tree after the compiled file is read in. It will no longer increase. The parsed storage structure only includes public part information such as header, vectorList, and a small amount of token and label information that may be used for jumps. In addition to the token tree, the entire compilation process only retains a small amount of public token information, greatly reducing memory usage. When compiling super large files, the memory usage is stable and compilation can be smoothly achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 A flowchart of the antlr tool of the present invention executed according to the method;
[0026] Figure 2 Generate a flow chart for the target compilation file of the present invention;
[0027] Figure 3 This is a diagram showing the final storage structure of the programcontext of the present invention;
[0028] Figure 4 It is the data flow guide diagram of the antlr parsing process of the present invention;
[0029] Figure 5 This is a data processing flow chart after ANTLR parsing of the present invention is completed. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] The embodiment of the present application provides a method based on antlr tool to support the compilation of super large source code files, which solves the problem of three memory collections generated in the traditional source code compilation process. If the source code is large and the number of tokens per line of source code is large, the memory will increase sharply. When compiling more than 5 million lines of source code, the memory will quickly increase to 10G, and the operating system will be stuck or even black screen.
[0032] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0033] The embodiment of the present invention discloses a method for supporting the compilation of super large source code files based on an antlr tool.
[0034] According to the attached Figure 1-Figure 5 As shown, the following steps are included;
[0035] S1. Source code parsing: start the antlr tool, read the source code file through the antlr runtime library, and perform source code parsing;
[0036] S2, token encoding, after the source code is parsed, it is stored in the antlr runtime library memory, and the visitor of the token tree generated by antlr is used to parse line by line, and the public type token is stored in the memory;
[0037] S3. Token judgment in the function: parse a large number of tokens in the function to determine their validity. If the current line statement is valid, encode the token in the current line and save it in a temporary file, and delete the token of the current statement from the memory. If the current line statement is invalid, the compilation ends, prompts an error, and returns.
[0038] S4. Judge the public part token in the memory. When the last line is parsed and transcoded, judge the public part token in the memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, prompts an error, and returns.
[0039] S5. Compile file output. The compilation result in the programcontext memory in antlr is directly written into the target compilation file. Then the temporary file is read and the code in the temporary file is added to the corresponding position of the target compilation file to complete the compilation of the super large source code file.
[0040] When the vistor traverses the tokens corresponding to each line of statements, the tokens will be encoded and stored in temporary files and token memory sets, and the encoded memory will no longer be retained. The content that needs to be compiled in the context is stored in the memory, which accounts for a small number of contents, such as jumps, calls to other functions, etc. When the parsing and compilation reaches the last statement, the temporary file is combined with other information to form a complete compiled file. The entire memory is mainly the memory size stored in the antlr parsing tree after the compiled file is read in, and the storage structure after parsing only includes public part information such as header, vectorList, and a small amount of token (label) information that may be used for jumps. In addition to the token tree, the entire compilation process only retains a small amount of public token information, which greatly reduces the memory usage. When compiling very large files, the memory usage is stable and the compilation can be smoothly implemented.
[0041] Preferably, the antlr optimizes the storage structure, and the final vector content only includes common part information such as header, vectorList, etc., and also includes token and label information for jump; while in the existing compilation method, the header,
[0042] Vectorlist is public token information, and the number is not large. However, one statement of the vector function corresponds to a normalVector structure. In the case of many statements, a large amount of memory will be occupied. In the subsequent encoding process, the encoding of all tokens for each statement is also stored in the memory. Therefore, the encoding process will also store a large amount of encoding in the memory for writing into the compiled target file, which will cause the entire memory to expand rapidly and make the program stuck and unresponsive.
[0043] Preferably, in step S1, when parsing the source code, the common part of the tokens parsed out of the token tree are stored in the programcontext memory, and at the same time, in the token tree, all the tokens for parsing function statements are stored in the normalvector temporary memory to form a token memory set, and this part of the tokens is directly deleted from the programcontext memory, and the encoding memory is no longer retained, and finally all the tokens in the normalvector temporary memory are encoded, and the encoding results are stored in a temporary file, and when the visitor of the token tree generated by antlr traverses the tokens corresponding to each line of statements, it is repeatedly executed, thereby reducing the generation of a memory set for each independent parsing function statement in the programcontext memory, thereby reducing the programcontext memory load.
[0044] Preferably, after completing all source code parsing, the antlr directly writes the compilation results in the programcontext memory into the target compilation file when processing the data therein, and writes the encoding results in the temporary file into the corresponding offset position of the target compilation file.
[0045] Preferably, in steps S3 and S4, when judging the token in the function and the public part token in the memory, the visitor traverses the token tree to read each line of source code in the function, and directly judges according to the settings of the grammar rule g4 file and other specific rules defined by the language. The judgment is successful if it conforms to the regular expression grammar. If the judgment is successful, the code is stored in a temporary file, and the token corresponding to the line of source code visited by the vistor is deleted in the memory, and the code of other lines of statements traversed subsequently is appended to the temporary file.
[0046] Preferably, after the steps S3 and S4 are completed, the token types belonging to the public part that may be called by other tokens are stored in the memory until the syntax is checked again at the end, and they are transcoded and written to the corresponding position of the compiled target file.
[0047] Preferably, in step S5, the specific steps of compiling the file output are as follows:
[0048] Step 1: Encode the public part and write the result into the target file;
[0049] Step 2: Write the split code between the common part and the function statement;
[0050] Step 3: Read the temporary file content and write it into the target file in binary form;
[0051] Step 4: Write the remaining necessary parts into the target file;
[0052] Step 5: Form the final target file.
[0053] When compiling more than 5 million lines of source code through this compilation method, the biggest difference from the existing compilation method is that the traditional source code compilation process generates three memory collections during the compilation process. When there are many tokens in each line of source code, the memory will increase sharply. Each token contains information such as line number, column number, type, name, etc., with a total size of about 120 bytes. The memory size increased by the visitor parsing a line of source code is 39*120=4680 Bytes = 4.5K. When compiling more than 5 million lines of source code, the memory quickly rises to 10G, causing the operating system to freeze or even go black. The essence of the problem is that antlr generates a large amount of temporary memory when accessing the token tree, and also generates a large amount of memory when compiling the source code according to the lexical structure. When vistor traverses the tokens corresponding to each line of statement, this method encodes these tokens and stores them in temporary files. The encoding memory of the token memory set will no longer be retained. When the content compiled in the context needs to be stored in the memory, such as jumping, calling other functions, etc., all tokens are stored in the normalvector temporary memory to form a token memory set, and this part t oken is directly deleted from the programcontext memory, the encoding memory is no longer retained, and finally all tokens in the normalvector temporary memory are encoded, and the encoding results are stored in a temporary file. When the visitor of the token tree generated by antlr traverses the token corresponding to each line of the statement, it is repeatedly executed. When the parsing and compilation reaches the last statement, the temporary file is combined with other information to form a complete compiled file. The entire memory is mainly the memory size stored in the antlr parsing tree after the compiled file is read in, and will not be increased additionally, so that the entire system requires very little content during operation, can run stably, and avoid system unresponsiveness and crashes.
[0054] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A method for supporting the compilation of super large source code files based on antlr tools, characterized in that: The steps include: S1. Source code parsing: start the antlr tool, read the source code file through the antlr runtime library, and perform source code parsing; S2, token encoding, after the source code is parsed, it is stored in the antlr runtime library memory, and the visitor of the token tree generated by antlr is used to parse line by line, and the public type token is stored in the memory; S3. Token judgment in the function: parse a large number of tokens in the function to determine their validity. If the current line statement is valid, encode the token in the current line and save it in a temporary file, and delete the token of the current statement from the memory. If the current line statement is invalid, the compilation ends, prompts an error, and returns. S4. Judge the public part token in the memory. When the last line is parsed and transcoded, judge the public part token in the memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, prompts an error, and returns. S5, compiled file output, the compilation result in the programcontext memory in antlr is directly written into the target compiled file, and then the temporary file is read, and the code in the temporary file is added to the corresponding position of the target compiled file to complete the compilation of the super large source code file; In the step S1, when the source code is parsed, the common part tokens parsed from the token tree are stored in the programcontext memory, and at the same time, all the tokens of the parsed function statements in the token tree are stored in the normalvector temporary memory to form a token memory set, and this part of the tokens is directly deleted from the programcontext memory, and the encoding memory is no longer retained, and finally all the tokens in the normalvector temporary memory are encoded, and the encoding results are stored in a temporary file, and when the visitor of the token tree generated by antlr traverses the tokens corresponding to each line of statements, it is repeatedly executed.
2. A method for supporting the compilation of super large source code files based on antlr tools according to claim 1, characterized in that: The antlr optimizes the storage structure, and the final vector content only includes the common part information of header and vectorList, and also includes the token and label information used for jumping.
3. A method for supporting the compilation of super large source code files based on antlr tools according to claim 2, characterized in that: After completing all source code parsing, the antlr processes the data therein and directly writes the compilation results in the programcontext memory into the target compilation file, and writes the encoding results in the temporary file into the corresponding offset position of the target compilation file.
4. A method for supporting the compilation of super large source code files based on antlr tools according to claim 1, characterized in that: In the steps S3 and S4, when judging the token in the function and the public part token in the memory, the visitor traverses the token tree to read each line of source code in the function, and directly judges according to the settings of the grammar rule g4 file and other specific rules defined by the language. If it conforms to the regular expression grammar, the judgment is successful. If the judgment is successful, the code is stored in a temporary file, and the token corresponding to the line of source code visited by the vistor is deleted in the memory, and the code of other lines of statements traversed subsequently is appended to the temporary file.
5. A method for supporting the compilation of super large source code files based on antlr tools according to claim 4, characterized in that: After the steps S3 and S4 are completed, the token types belonging to the public part that will be called by other tokens will be stored in the memory until the syntax is checked again, transcoded, and written to the corresponding position of the compiled target file.
6. A method for supporting the compilation of super large source code files based on antlr tools according to claim 1, characterized in that: In step S5, the specific steps of compiling the file output are as follows: Step 1: Encode the public part and write the result into the target file; Step 2: Write the split code between the common part and the function statement; Step 3: Read the temporary file content and write it into the target file in binary form; Step 4: Write the remaining necessary parts into the target file; Step 5: Form the final target file.
Citation Information
Patent Citations
ANTLR (Another Tool for Language Recognition)-based importing method for LDF (Log Data File) of description file of LIN (Local Interconnect Network)
CN102521291A
Cross-platform multi-language compiling method and ultra-light Internet-of-Things virtual machine
CN113703779A