Method for supporting compilation of ultra-large source code file on basis of antlr tool

Abstract

The present invention relates to the field of compilation of source code files. Disclosed is a method for supporting the compilation of an ultra-large source code file on the basis of an ANTLR tool. In the method for supporting the compilation of an ultra-large source code file on the basis of an ANTLR tool, when traversing tokens corresponding to each row of statements, a visitor encodes the tokens and stores the encoded tokens in a temporary file, and a token memory set and an encoding memory are no longer retained; when the last statement is parsed and compiled, the temporary file is combined with other information to form a complete compiled file; the size of the total memory mainly depends on a memory occupied in an ANTLR parse tree after the compiled file is read into the memory; and a parsed storage structure contains only information of public sections such as header and vectorList, and a small amount of token and label information for jumping, such that the overall memory footprint is greatly reduced. When an ultra-large file is compiled, the memory footprint remains stable, thereby enabling smooth compilation.

Description

A method for supporting the compilation of extremely large source code files based on the ANTLR tool Technical Field

[0001] This invention relates to the field of source code file compilation technology, specifically a method for supporting the compilation of extremely large source code files based on the ANTLR tool. Background Technology

[0002] ANTLR is a powerful parser generator that supports the generation of lexical analyzers and parsers for custom programming languages. ANTLR's main uses include implementing data loaders, language interpreters, and language translators. It parses files based on automatically generated parsing trees and is primarily geared towards the semiconductor testing field.

[0003] The AntLR tool generates a language's lexical tree, parse tree, and access tree source code project based on the language's syntax rules (.g4 file). This project is then used to compile the language's source code. Since the visitor stores each node in memory while traversing the tree, the memory usage increases proportionally if the source file is large. Finally, the node information is retrieved from memory and compiled into a file. This traditional source code compilation process generates three memory sets. If the source code is large, and each line of source code contains many tokens, memory usage will increase dramatically. When compiling source code exceeding 5 million lines, memory usage can quickly reach 10GB, causing the operating system to freeze or even crash. The root cause of this problem is that AntLR generates a large amount of temporary memory during token tree access and also during lexical compilation. Therefore, a method based on the AntLR tool to support the compilation of extremely large source files is provided to address these issues. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for compiling extremely large source code files based on the ANTLR tool, solving the problem of three memory aggregations occurring in the traditional source code compilation process. If the source code is large, and the number of tokens per line of source code is high, memory usage will increase dramatically. When compiling source code exceeding 5 million lines, memory usage can quickly rise to 10GB, causing the operating system to freeze or even resulting in a black screen.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for supporting the compilation of extremely large source code files based on the ANTLR tool, comprising the following steps;

[0006] S1. Source code analysis: Start the ANTLR tool, read the source code file through the ANTLR runtime library, and perform source code analysis.

[0007] S2, token encoding, after source code parsing, is stored in the ANTLR runtime library memory, and the visitor of the token tree generated by ANTLR parses it line by line, and the public type token is stored in memory;

[0008] S3. Function-based token judgment: For a large number of tokens of various types within a function, the validity of the tokens is determined. If the current line statement is valid, the token of the current line is encoded, stored in a temporary file, and the token of the current line statement is deleted from memory. If the current line statement is invalid, the compilation ends, an error is displayed, and the process returns.

[0009] S4. Check the token in the common part of memory. When the last line is parsed and transcoded, check the token in the common part of memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, an error is displayed, and the process returns.

[0010] S5. Compilation file output: The compilation results in the ProgramContext memory of ANTLR are directly written to the target compilation file. Then, the temporary file is read and the encoding in the temporary file is added to the corresponding position in the target compilation file to complete the compilation of the very large source code file.

[0011] Preferably, by optimizing the storage structure, the final vendor content of the ANTLR only includes common information such as header and vectorList, and also includes token and label information for navigation.

[0012] Preferably, in step S1, when parsing the source code, the common part of the tokens parsed from the token tree is stored in the program context memory. At the same time, all tokens for parsing function statements in the token tree are stored in the normal vector temporary memory, forming a token memory set. This part of the tokens is directly deleted from the program context memory, and the encoding memory is no longer retained. Finally, all tokens in the normal vector temporary memory are encoded, and the encoding result is stored in a temporary file. This process is repeated when the visitor of the token tree generated by ANTLR traverses the tokens corresponding to each line of statements.

[0013] Preferably, after completing all source code parsing, when processing the data, the ANTLR directly writes the compilation results in the ProgramContext memory into the target compiled file, and simultaneously writes the encoding results in the temporary file into the corresponding offset position of the target compiled file.

[0014] Preferably, in steps S3 and S4, when judging the token within the function and the token in the common part of memory, the visitor traverses the token tree to read each line of source code within the function. The judgment is made directly according to the syntax rules set in the g4 file and other specific rules defined by the language. If the regular expression syntax is met, the judgment is successful. If the judgment is successful, the encoding is stored in a temporary file, and the token corresponding to the line of source code accessed by the visitor is deleted from memory. The encoding of subsequent traversals of other lines of statements is appended to the temporary file.

[0015] Preferably, after steps S3 and S4 are completed, the public part of the token type that may be called by other tokens is stored in memory until the syntax is checked again, and then it is transcoded and written to the corresponding location in the compiled target file.

[0016] Preferably, in step S5, the specific steps for compiling the file output are as follows;

[0017] Step 1: Encode the common parts and write the results to the target file;

[0018] Step 2: Write a separator between the common part and the function statement;

[0019] Step 3: Read the contents of the temporary file and write them to the target file in binary format;

[0020] Step 4: Write the remaining necessary parts into the target file;

[0021] Step 5: Create the final target file.

[0022] This invention discloses a method for supporting the compilation of extremely large source code files based on the ANTLR tool, which has the following beneficial effects:

[0023] 1. This method, based on the ANTLR tool, supports the compilation of extremely large source code files. When the visitor traverses the tokens corresponding to each line of statements, these tokens are encoded and stored in a temporary file or token memory set. The encoded memory is no longer retained. Content requiring contextual compilation is stored in memory; this portion is small, such as jumps and calls to other functions. When the compilation reaches the last statement, the temporary file is combined with other information to form a complete compiled file. The total memory usage is mainly the size of the compiled file stored in the ANTLR parse tree, and does not increase further. The parsed storage structure only includes common information such as headers and vectorLists, as well as a small amount of tokens and labels that may be used for jumps. This ensures that the entire compilation process retains only a small amount of common token information besides the token tree, significantly reducing memory usage. When compiling extremely large files, memory usage remains stable, allowing for successful compilation. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 is a flowchart of the ANTLR tool of the present invention being executed according to the present convenient method;

[0026] Figure 2 is a flowchart of the target compilation file generation process of the present invention;

[0027] Figure 3 is a diagram showing the final storage structure of the programcontext of the present invention;

[0028] Figure 4 is a data flow diagram of the ANTLR parsing process of the present invention;

[0029] Figure 5 is a flowchart of the data processing after the ANTLR parsing is completed according to the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This application provides a method for compiling extremely large source code files based on the ANTLR tool, solving the problem of three memory aggregations occurring in the traditional source code compilation process. If the source code is large, and the number of tokens per line is high, memory usage will increase dramatically. When compiling source code with more than 5 million lines, memory usage can quickly rise to 10GB, causing the operating system to freeze or even resulting in a black screen.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] This invention discloses a method for supporting the compilation of extremely large source code files based on the ANTLR tool.

[0034] The steps are as shown in Figures 1-5;

[0035] S1. Source code analysis: Start the ANTLR tool, read the source code file through the ANTLR runtime library, and perform source code analysis.

[0036] S2, token encoding, after source code parsing, is stored in the ANTLR runtime library memory, and the visitor of the token tree generated by ANTLR parses it line by line, and the public type token is stored in memory;

[0037] S3. Function-based token judgment: For a large number of tokens of various types within a function, the validity of the tokens is determined. If the current line statement is valid, the token of the current line is encoded, stored in a temporary file, and the token of the current line statement is deleted from memory. If the current line statement is invalid, the compilation ends, an error is displayed, and the process returns.

[0038] S4. Check the token in the common part of memory. When the last line is parsed and transcoded, check the token in the common part of memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, an error is displayed, and the process returns.

[0039] S5. Compilation file output: The compilation results in the ProgramContext memory of ANTLR are directly written to the target compilation file. Then, the temporary file is read and the encoding in the temporary file is added to the corresponding position in the target compilation file to complete the compilation of the very large source code file.

[0040] By traversing each line of statements using the visitor, the tokens are encoded and stored in a temporary file, a token memory set. The encoded memory is no longer retained. Content requiring contextual compilation is stored in memory; this portion is small, such as jumps and calls to other functions. When the final statement is parsed and compiled, the temporary file is combined with other information to form a complete compiled file. The total memory usage is primarily the size of the compiled file stored in the ANTLR parse tree, without increasing the memory footprint. The parsed storage structure only includes common information such as headers and vectorLists, and a small amount of token (label) information that may be used for jumps. This ensures that the entire compilation process retains only a small amount of common token information besides the token tree, significantly reducing memory usage. Even when compiling very large files, memory usage remains stable, allowing for successful compilation.

[0041] Preferably, the ANTLR optimizes the storage structure so that the final vendor content only includes common information such as header and vectorList, as well as token and label information for navigation; while existing compilation methods parse the token tree storage structure, including header,

[0042] VectorList contains public token information, which is not numerous. However, each statement in the Vector function corresponds to a NormalVector structure. With many statements, this will consume a large amount of memory. Furthermore, during the subsequent encoding process, the encoding of all tokens for each statement is also stored in memory. Therefore, the encoding process will also store a large amount of encoding in memory for writing to the compilation target file, resulting in a rapid expansion of the entire memory and causing the program to freeze and become unresponsive.

[0043] Preferably, in step S1, when parsing the source code, the common part of the tokens parsed from the token tree is stored in the program context memory. At the same time, all tokens for parsing function statements in the token tree are stored in the normal vector temporary memory, forming a token memory set. This part of the tokens is directly deleted from the program context memory, and the encoding memory is no longer retained. Finally, all tokens in the normal vector temporary memory are encoded, and the encoding result is stored in a temporary file. When the visitor of the token tree generated by ANTLR traverses the tokens corresponding to each line of statements, this process is repeated to reduce the generation of a memory set for each independent parsing function statement in the program context memory, thereby reducing the memory load of the program context.

[0044] Preferably, after completing all source code parsing, when processing the data, the ANTLR directly writes the compilation results in the ProgramContext memory into the target compiled file, and simultaneously writes the encoding results in the temporary file into the corresponding offset position of the target compiled file.

[0045] Preferably, in steps S3 and S4, when judging the token within the function and the token in the common part of memory, the visitor traverses the token tree to read each line of source code within the function. The judgment is made directly according to the syntax rules set in the g4 file and other specific rules defined by the language. If the regular expression syntax is met, the judgment is successful. If the judgment is successful, the encoding is stored in a temporary file, and the token corresponding to the line of source code accessed by the visitor is deleted from memory. The encoding of subsequent traversals of other lines of statements is appended to the temporary file.

[0046] Preferably, after steps S3 and S4 are completed, the public part of the token type that may be called by other tokens is stored in memory until the syntax is checked again, and then it is transcoded and written to the corresponding location in the compiled target file.

[0047] Preferably, in step S5, the specific steps for compiling the file output are as follows;

[0048] Step 1: Encode the common parts and write the results to the target file;

[0049] Step 2: Write a separator between the common part and the function statement;

[0050] Step 3: Read the contents of the temporary file and write them to the target file in binary format;

[0051] Step 4: Write the remaining necessary parts into the target file;

[0052] Step 5: Create the final target file.

[0053] The biggest difference between this compilation method and existing methods when compiling source code exceeding 5 million lines lies in the fact that traditional source code compilation generates three memory sets during the compilation process. When the number of tokens per line of source code is large, memory usage increases dramatically. Each token contains information such as line number, column number, type, and name, with a total size of approximately 120 bytes. The memory increase for the visitor parsing one line of source code is 39 * 120 = 4680 bytes = 4.5K. When compiling source code exceeding 5 million lines, memory usage quickly rises to 10G, causing the operating system to freeze or even display a black screen. The root cause of this problem is that ANTLR generates a large amount of temporary memory during token tree access and also generates a large amount of memory during lexical compilation of the source code. In contrast, this method encodes these tokens and stores them in a temporary file when the visitor traverses the tokens corresponding to each line of code. The memory usage for encoding the token memory set is significantly reduced. No longer retained, when content requiring contextual compilation is stored in memory, such as jumps or calls to other functions, all tokens are stored in the temporary memory of the normal vector, forming a token memory set. This part of the tokens is directly deleted from the program context memory. The encoded memory is no longer retained. Finally, all tokens in the temporary memory of the normal vector are encoded, and the encoding results are stored in a temporary file. When the visitor of the token tree generated by ANTLR traverses the tokens corresponding to each line of statement, this process is repeated. When the last statement is parsed and compiled, the temporary file is combined with other information to form a complete compiled file. The total memory is mainly the size of the memory in the ANTLR parsing tree after the compiled file is read in, and no additional memory is added. This makes the entire system require very little memory during operation, enabling stable operation and avoiding system unresponsiveness and crashes.

[0054] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for supporting the compilation of extremely large source code files based on the ANTLR tool, characterized in that, Includes the following steps; S1. Source code analysis: Start the ANTLR tool, read the source code file through the ANTLR runtime library, and perform source code analysis. S2, token encoding, after source code parsing, is stored in the ANTLR runtime library memory, and the visitor of the token tree generated by ANTLR parses it line by line, and the public type token is stored in memory; S3. Function-based token judgment: For a large number of tokens of various types within a function, the validity of the tokens is determined. If the current line statement is valid, the token of the current line is encoded, stored in a temporary file, and the token of the current line statement is deleted from memory. If the current line statement is invalid, the compilation ends, an error is displayed, and the process returns. S4. Check the token in the common part of memory. When the last line is parsed and transcoded, check the token in the common part of memory. If the token is valid, encode the token and write it into the target compilation file. If the token is invalid, the compilation ends, an error is displayed, and the process returns. S5. Compilation file output: The compilation results in the ProgramContext memory of ANTLR are directly written to the target compilation file. Then, the temporary file is read and the encoding in the temporary file is added to the corresponding position in the target compilation file to complete the compilation of the very large source code file. In step S1, when parsing the source code, the common part of the tokens parsed from the token tree is stored in the programcontext memory. At the same time, all tokens for parsing function statements in the token tree are stored in the normalvector temporary memory, forming a token memory set. This part of the tokens is directly deleted from the programcontext memory, and the encoding memory is no longer retained. Finally, all tokens in the normalvector temporary memory are encoded, and the encoding result is stored in a temporary file. This process is repeated when the visitor of the token tree generated by ANTLR traverses the tokens corresponding to each line of statements.

2. The method for supporting the compilation of extremely large source code files based on the ANTLR tool according to claim 1, characterized in that: By optimizing the storage structure, the final vector content of the ANTLR only includes the common information of the header and vectorList, as well as the token and label information used for navigation.

3. The method for supporting the compilation of extremely large source code files based on the ANTLR tool according to claim 2, characterized in that: After completing the parsing of all source code, when processing the data, ANTLR directly writes the compilation results in the program context memory into the target compiled file, and at the same time writes the encoding results in the temporary file into the corresponding offset position of the target compiled file.

4. The method for supporting the compilation of extremely large source code files based on the ANTLR tool according to claim 1, characterized in that: In steps S3 and S4, when judging the token within the function and the token in the common part of memory, the visitor traverses the token tree to read each line of source code within the function. The judgment is made directly according to the syntax rules set in the g4 file and other specific rules defined by the language. If the regular expression syntax is correct, the judgment is successful. If the judgment is successful, the encoding is stored in a temporary file, and the token corresponding to the line of source code accessed by the visitor is deleted from memory. The encoding of other lines of statements traversed subsequently is appended to the temporary file.

5. A method for supporting the compilation of extremely large source code files based on the ANTLR tool according to claim 4, characterized in that: After steps S3 and S4 are completed, the public part of the token type that will be called by other tokens will be stored in memory until the syntax is checked again, the token is transcoded, and written to the corresponding location in the compiled target file.

6. The method for supporting the compilation of extremely large source code files based on the ANTLR tool according to claim 1, characterized in that: In step S5, the specific steps for compiling the file output are as follows; Step 1: Encode the common parts and write the results to the target file; Step 2: Write a separator between the common part and the function statement; Step 3: Read the contents of the temporary file and write them to the target file in binary format; Step 4: Write the remaining necessary parts into the target file; Step 5: Create the final target file.

Images