Function level incremental compiling method and system for software project

Abstract

The invention discloses a function-level incremental compiling method and system for a software project, and the method comprises the steps: recognizing a function with text change, and extracting a symbol used in a change code; performing compilation-dependent propagation on the CDG of the old source code based on a symbol used in the change code, and identifying a semantically changing function; deleting functions which do not change semantically to generate a part of source codes, and then compiling to generate a part of binary files; the dependency relationship between symbols is extracted from the old binary file to generate a BDG, and derived binary codes needing to be reused are recognized; for the function of which the main body is deleted, binary dependency propagation is executed on the BDG, and symbols and data segments which need to be multiplexed are determined; and combining symbols needing to be multiplexed and part of data segments in the old binary file into part of the binary file to form a complete binary file. According to the method, only the function with changed semantics needs to be recompiled according to the compiling result before function level reuse.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a function-level incremental compilation method and system for a software project. Background Art

[0002] During the development cycle of a C / C++ project, developers need to repeatedly modify and compile the project's source files. Even if only a small portion of the code is changed, existing C / C++ compilers still need to spend a lot of time recompiling the entire source file, resulting in a large amount of redundant compilation. The key to solving the redundant compilation problem is to reuse previously generated compilation results. However, existing methods can only reuse previous compilation results at the file level, resulting in a high time cost for recompilation.

[0003] Figure 1 This example shows how existing C / C++ compilers work when faced with a minor code change. The source file contains 1,000 functions, and the change modifies only one of them, adding a print statement on line 9. Unfortunately, the existing compiler needs to recompile all the functions in this file, resulting in 99.9% (999 / 1,000) of the functions being compiled redundantly. This inefficiency is further exacerbated when the redundant compilation spreads to other source files, thereby reducing the overall performance. In the given example, each function instantiates a template class vector with different types. Such instantiations significantly increase the code size of the recompilation, resulting in a high time cost.

[0004] Solving the redundant compilation problem mentioned above is a very difficult task. Existing methods cannot effectively handle Figure 1The main reason for the incremental compilation scenarios in the shown scenarios is that they are limited to reusing previous compilation results at the file level. One class of methods focuses on reusing old binaries for source files whose semantics remain unchanged. This includes CCACHE(CCache.CCache.https: / / ccache.dev / ,2024.[Online; accessed July-2024].), CHASH(Christian Dietrich, Valentin Rothberg, Ludwig Füracker, Andreas Ziegler, and Daniel Lohmann.cHash: Detection of redundant compilations via ast hashing.In2017USENIX Annual Technical Conference, pages 527–538, 2017. https: / / www.usenix.org / conference / atc17 / technical-essions / presentation / dietrich.) and ABC (Ying Zhang, Yanyan Jiang, Chang Xu, Xiaoxing Ma, and Ping Y u.ABC: Accelerated building of c / c++projects.In 2015Asia-Pacific SoftwareEngineering Conference (APSEC), pages 182–189, 2015. https: / / doi.org / 10.1109 / APSEC.2015.27.), which differ only in the method of determining the semantic equivalence of source files. These methods range from comparing MD5 hashes to comparing abstract syntax tree (AST) hashes to analyzing whether the modified code is actually used by the source file. However, semantic changes in source files (such as Figure 1As shown in ( ), these methods are invalid when reusing the final compilation results. Another type of method includes precompiling header files (LLVM. Precompiled header. https: / / clang.llvm.org / docs / PCHInternals.html, 2024. [Online; accessed July-2024].) or modules (CPP Reference. Modules. https: / / en.cppreference.com / w / cpp / language / modules, 2024. [Online; accessed July-2024].) into ASTs for reuse in subsequent compilations. Although this method can alleviate the need for header files (such as Figure 1 iostream and vector in the example shown, it still requires recompiling the source file. In addition, minor code changes in header files or modules may also destroy the effectiveness of such methods. Summary of the invention

[0005] The purpose of the present invention is to address the defects and shortcomings in the prior art and to propose a function-level incremental compilation method and system for software projects, which can reuse previous compilation results at the function level and only need to recompile functions with changed semantics, thereby alleviating the problem of redundant compilation.

[0006] The technical solution of the present invention is as follows.

[0007] In one aspect, a function-level incremental compilation method for a software project includes:

[0008] The symbol change analysis step identifies functions with text changes and extracts the symbols used in the changed code through lexical analysis;

[0009] The compilation dependency propagation step performs compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and the symbols used in the changed code to identify the functions that have changed semantically;

[0010] A selective compilation step that prunes the bodies of semantically unchanged functions, generating only partial source code that is then compiled to produce partial binaries;

[0011] The binary dependency propagation step extracts the dependencies between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived binary code that needs to be reused; then, for the function whose main body is deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused;

[0012] In the binary reuse step, the symbols and some data segments that need to be reused in the old binary file are merged into the partial binary file to form a complete binary file.

[0013] Preferably, the symbol change analysis step specifically includes:

[0014] Preprocess the source code and identify the text differences between the new and old preprocessed codes; if the text of the function body changes, it is determined that the semantics of the function has changed;

[0015] Lexical analysis is performed to extract identifiers in the textual change code, and it is assumed that symbols with the same names as these identifiers will be used by the change code.

[0016] Preferably, the compilation dependency graph CDG includes nodes and edges; wherein the nodes represent symbols in the source code; and the edges are used to describe the dependency relationships between symbols, including resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies.

[0017] Preferably, the compilation dependency propagation specifically includes:

[0018] Starting from the functions with changed text and the symbols used by the changed code, recursively traverse the resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies on the CDG;

[0019] After the propagation is completed, if the function meets one of the following two conditions, the semantics of the function is considered unchanged; the two conditions include:

[0020] This function has not been accessed during the propagation process;

[0021] The function was accessed during propagation, but the function's text was not changed, the function was not a template function, and the binary code of the function existed in the previous binary.

[0022] Preferably, the selective compilation step further includes: during the compilation process, using semantic analysis to eliminate lexical errors and dependencies.

[0023] Preferably, the binary dependency graph BDG includes nodes and edges, wherein the nodes represent binary symbols and sections; and the edges are used to describe the dependency relationship between binary symbols and segments, including symbol relocation dependency and binary section dependency.

[0024] Preferably, the binary dependency propagation step adopts breadth-first search; starting from the function with unchanged semantics, symbol relocation dependencies and section dependencies are recursively traversed on BDG; if the traversed symbols or sections do not exist in the streamlined binary code, they will be reused.

[0025] Preferably, in the binary multiplexing step, the binary multiplexing includes real-name symbol multiplexing and anonymous symbol multiplexing;

[0026] The symbol name of the real-name symbol is used as a unique identifier so that the named symbol can be matched between the old binary file and the partial binary file; in addition, the section derived from the named symbol also has a unique identifier; if the named symbol or section to be reused does not exist in the partial binary file, these symbols and sections are created, the symbol table and the section header table are updated, and the symbol name is added to the string table; if these symbols and sections already exist in the partial binary file, the corresponding symbol and section indexes are adjusted to point to the existing symbols and sections;

[0027] For read-only data, the compiler generates a randomly named read-only data section and an anonymous symbol. In the binary, anonymous symbols are identified by section offsets or by using random symbol names. For anonymous symbols that need to be reused, a new anonymous symbol and a new read-only data section are directly created in some binary files. The binary data stored by the anonymous symbol will be reused in the new read-only data section when needed.

[0028] In another aspect, a function-level incremental compilation system for a software project includes:

[0029] The symbol change analysis module is used to identify functions with text changes and extract the symbols used in the changed code through lexical analysis;

[0030] A compilation dependency propagation module is used to perform compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and symbols used in the changed code to identify the functions that have changed semantically;

[0031] A selective compilation module that prunes the bodies of semantically unchanged functions, generating only partial source code that is then compiled to produce partial binaries;

[0032] The binary dependency propagation module is used to extract the dependencies between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived symbols and partial data segments that need to be reused; then, for the function whose main body is deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused;

[0033] The binary reuse module is used to merge the symbols and partial data segments that need to be reused in the old binary file into the partial binary file to form a new and complete binary file.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The present invention can reuse previous compilation results at the function level, and only needs to recompile functions with changed semantics, which can alleviate the problem of redundant compilation;

[0036] (2) The present invention introduces two symbolic dependency representations, namely, compilation dependency graph (CDG) and binary dependency graph (BDG), at the source code and binary code levels. CDG and BDG help to better identify functions with changed semantics and smoothly reuse binary codes related to functions with unchanged semantics, thereby ensuring the correctness of function-level incremental compilation. Among them, the dependency graph (CDG) captures the dependencies between symbols at the source code level. By utilizing CDG, it is possible to well identify functions whose bodies remain unchanged but whose semantics have changed. The binary dependency graph (BDG) captures symbolic dependencies at the binary code level. Using BDG, it is possible to extract binary codes of functions and other dependent symbols (e.g., derived read-only data, template instantiations, etc.) whose semantics have not changed from old binary files. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A diagram of how existing C / C++ compilers work when faced with minor code changes;

[0039] Figure 2 Schematic diagram of the function for identifying semantic changes;

[0040] Figure 3 A flow chart of a function-level incremental compilation method for a software project according to an embodiment of the present invention;

[0041] Figure 4 is an overall flow chart of the function-level incremental compilation method according to an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of compilation dependency propagation according to an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of binary dependency propagation according to an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of a multiplexed binary code according to an embodiment of the present invention;

[0045] Figure 8It is a schematic diagram of the acceleration distribution on Clang implemented by IClang and ABC according to an embodiment of the present invention;

[0046] Fig. 9 This is a structural diagram of a function-level incremental compilation system for a software project according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Implementing function-level incremental compilation is not easy. There are two problems that need to be solved.

[0049] (1) How to correctly identify functions with semantic changes

[0050] Failure to compile a function whose semantics have changed will result in incorrect compilation results. Conversely, ignoring a function whose semantics remain unchanged will not result in incorrect compilation results. Therefore, correctly identifying semantic changes is crucial. It is also important to note that a function whose body has not changed will not necessarily retain its original semantics. This is because the side effects of a semantically changed function can propagate to other functions, even if their code remains unchanged. For example, Figure 2 As shown in the figure, the updated source code introduces a change that requires code to be generated to instantiate the template test with a floating-point type (test). However, this particular generated code does not exist in the previous binary code. If the function body of test is deleted, the binary code of test cannot be generated correctly, resulting in an undefined symbol error. In addition, compiler features such as unused symbol optimization and implicit code generation can also affect the semantics of the function.

[0051] (2) How to correctly reuse the binary code of a function

[0052] Directly reusing the binary code of a function that has not changed semantics is not enough to ensure correct compilation. The disadvantage is that even if the source code of a function has not changed, its internal source code may still cause additional source code to be generated. For example, the location where a template class is used may trigger the instantiation of the template class. Therefore, when deleting a function that has not changed semantics, the binary code of any derived code must also be reused. Figure 2 For example, if the body of function foo is deleted, it will prevent test <int>Reusing only the binary code of foo will result in calling test <int>In addition, deleting a function may also cause the generation of related read-only data and exception tables to be omitted.

[0053] See also Figure 3 As shown, this embodiment discloses a function-level incremental compilation method for a software project that solves the above two problems, including the following steps:

[0054] S301, symbol change analysis step, identifies functions with text changes, and extracts symbols used in the changed code through lexical analysis.

[0055] Specifically, first identify which functions' semantics are directly affected by the changed code. To this end, preprocess the source code and identify the text differences between the old and new preprocessed codes. If the text of the function body changes, it is conservatively assumed that the semantics of the function has changed. In order to determine whether the change in the function affects the semantics of other functions, it is necessary to obtain the symbolic dependencies in the new source code. In order to avoid the high cost of symbolic dependency analysis, the present invention reuses the previously generated CDG. However, the old CDG lacks the symbolic dependencies brought by the changed code. Therefore, lexical analysis is performed to extract identifiers in the textual change code, and it is conservatively assumed that symbols with the same names as these identifiers will be used by the changed code.

[0056] After symbol change analysis, two basic inputs can be provided for compilation dependency propagation: the changed functions and the symbols used by the changed code. Figure 4 As shown, the changed functions test1 and main can be identified through text differences, and the symbols used in the changed code, such as Derived, Base1, test, A, foo, temp, Base2, etc., can be obtained through lexical analysis.

[0057] S302, a compilation dependency propagation step, performs compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and symbols used in the changed code to identify functions that have changed semantically.

[0058] Specifically, we first discuss which features of C / C++ compilers trigger the propagation of semantic changes, then design a Compilation Dependency Graph (CDG) to accurately encode these features. In addition, we design a compilation dependency propagation method to identify semantically changed functions.

[0059] (1) Semantic Impact of Compiler Features

[0060] In C / C++ compilers, there are two features that affect the semantics of literal unchanged functions: unused symbol optimization and template instantiation.

[0061] 1) Unused symbol optimization: The C / C++ compiler automatically eliminates unused WEAK and LOCAL symbols to improve compilation efficiency. The compiler usually compiles member functions as WEAK symbols and symbols in anonymous namespaces as LOCAL symbols. If a WEAK or LOCAL function is not used in the old source code but is used in the new source code, trimming its function body will result in an undefined symbol error. For example, in Figure 4 In the new source code, A::foo is used. Since symbol optimization is not used, its binary code is not generated in the old binary file. If the A::foo function body with unchanged text is trimmed, its binary code cannot be correctly generated or reused from the old binary file, resulting in a symbol undefined error.

[0062] 2) Template instantiation: New source code may instantiate a template function with a new type. If the body of the template function is deleted, the template instantiation will not be generated correctly, resulting in an undefined symbol error. Figure 4 In the new source code, the template function temp (temp) is instantiated using the floating-point type. <float>), and the instantiation does not exist in the old binary code. If the function body of temp is deleted, temp <float>The binary code will not be generated correctly, resulting in undefined symbol errors.

[0063] (2) Compile dependency graph

[0064] A Compilation Dependency Graph (CDG) is designed to identify functions whose semantics remain unchanged. In CDG, nodes represent symbols in the source code, and edges describe the dependencies between symbols, including resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies.

[0065] 1) Resolved symbol dependencies: This type of dependency describes the reference relationship between symbols. For example, in Figure 5 In the example, function main references the template function Derived::test, so a resolved symbol dependency is established between main and Derived::test. By leveraging resolved reference dependencies, it is possible to determine whether WEAK / LOCAL / TEMPLATE functions will be used by the changed code, thereby avoiding pruning these functions incorrectly.

[0066] 2) Unresolved symbol dependencies: The C++ compiler represents symbols used in templates as unresolved symbols, whose specific destination is determined by the type of template instantiation. Figure 4 For example, in the old source code, the template function Derived::test uses the unresolved symbol impl. Since Derived::test is instantiated with type Base1, the target of impl will be resolved to Base1::impl. When the instantiation type of the template changes, the target of its unresolved symbol will also change. Figure 4 In , the new source code introduces a Derived::test instantiated with the Base2 type, and the target of impl will be resolved to Base1::impl and Base2::impl. However, relying solely on lexical analysis, it is impossible to accurately identify which new types of templates are instantiated in the new source code, which may lead to missing dependencies. In addition, if the missing dependencies are cutting edges from the changed code to WEAK / LOCAL / template functions, these functions may be incorrectly pruned. For example, the missing dependency from Derived::test to Base2::impl will cause the WEAK function Base2::impl to be incorrectly pruned. To ensure accurate identification of functions with unchanged semantics, it is conservatively assumed that the unresolved symbols may refer to any symbol with the same name. For example, in Figure 5 , conservatively assuming that Derived::test may reference Base1::impl and Base2::impl.

[0067] 3) Implicit symbol dependencies: In C++, constructors, destructors, operator functions, and virtual functions can be used implicitly. For example, the addition operator may implicitly call an overloaded addition function. To prevent implicit symbol dependencies in the code from being missed, a virtual node pointing to these symbols is created, and it is conservatively assumed that these symbols are always used.

[0068] (3) Compilation dependency propagation

[0069] Taking the textually changed functions, the symbols used by the changed code, the old CDG, and the symbol table of the old binary as input, a compilation dependency propagation method is designed to identify the semantically unchanged functions. The method essentially performs a breadth-first search on the old CDG. Starting from the textually changed functions and the symbols used by the changed code, the method recursively traverses the resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies on the CDG. After the propagation, if the function meets any of the following conditions, the semantics of the function can be considered unchanged:

[0070] The function was not accessed during the propagation process.

[0071] The function was accessed during propagation, its text was not changed, it is not a template function, and its binary code existed in the previous binary.

[0072] by Figure 5 For example, there are textual changes in test1 and main functions, while Derived::test, A::foo, and temp are used by the changed code. Starting from these functions, the old CDG is traversed and two functions with unchanged semantics are identified: test2 and Base::impl. test2 is not accessed during the propagation process. Although the textually unchanged function Base::impl is accessed during the propagation process, it is not a template function and exists in the previous binary (ZN5Base14implEv). Therefore, their semantics remain unchanged.

[0073] S303, a selective compilation step, deletes the body of the function that has not changed semantically, generates only part of the source code, and then compiles to generate part of the binary file.

[0074] Specifically, by deleting function bodies whose semantics have not changed, the amount of code that needs to be compiled can be significantly reduced, thereby reducing redundant compilation overhead. In addition, since the number of functions that need to be recompiled is reduced, semantic analysis no longer incurs significant overhead. Therefore, during the compilation process, semantic analysis is used to eliminate overly conservative dependencies introduced by lexical analysis. For example, an identifier that originally represents a class may be incorrectly associated with a function of the same name during lexical analysis, and this incorrect association will be eliminated by subsequent semantic analysis. This ensures that the CDG used for the next incremental compilation is accurate. Figure 4 For example, the bodies of functions Base1::impl and test2 whose semantics remain unchanged are deleted, and the CDG is updated through semantic analysis to add dependencies from main to temp and from main to A::foo.

[0075] In summary, the above S301 to S303 of this embodiment solves the problem of correctly identifying functions with semantic changes, and can accurately identify functions with semantic changes.

[0076] The Compilation Dependency Graph (CDG) designed in the above steps describes the dependencies between symbols in the source code. By traversing the CDG, it is possible to accurately identify which functions have changed in semantics. As a result, the bodies of functions whose semantics have not changed can be cut, thereby reducing redundant compilation overhead.

[0077] Building CDG is critical to identifying semantically changed functions. Therefore, the build time cost of CDG should be significantly lower than the time cost of recompiling the entire program. Since CDG captures symbolic dependencies and represents specific aspects of program semantics, a potential solution is to build CDG with semantic analysis during the compilation of the new version of the source file. However, it should be noted that semantic analysis introduces significant time overhead. The time cost of various analysis phases in five C / C++ projects in debug mode was measured in Clang, a well-known C / C++ compiler. On average, 78.8% of the compilation time was consumed in the front end, of which syntax and semantic analysis accounted for 68.0% of the total compilation time. This suggests that if CDG builds rely on syntax and semantic analysis during compilation, there may be limited room for optimizing incremental compilation performance.

[0078] The method of incrementally constructing a CDG of the present invention can reduce the time overhead involved in constructing a CDG. Specifically, the method of incrementally constructing a CDG utilizes the CDG generated during the previous compilation, which corresponds to the old version of the source file. A lightweight analysis technology based mainly on lexical analysis is designed, which conservatively identifies new symbols and symbol dependencies derived from the changed code. By temporarily updating the CDG of the old version of the source file with the newly extracted symbols and dependencies, the functions that have changed semantically can be accurately identified by traversing the updated CDG. This makes it possible to effectively delete functions that have not changed semantics and focus on compiling functions that have changed semantics. In addition, the temporarily updated CDG is refined based on semantic analysis during incremental compilation. This refinement ensures the generation of an accurate CDG for reuse in future incremental analysis.

[0079] See also Figure 2 As shown. By reusing the previously generated CDG, the dependency from function main to function foo, and the dependency from function foo to template function test can be identified. Through conservative lexical-based analysis, the dependency from function main to template function test can be further identified. Next, compilation dependency propagation will be performed on the CDG. According to the propagation strategy, it can be determined that the semantics of function main and template function test have changed, while the semantics of function foo remains unchanged. Therefore, the function body of foo can be safely deleted to reduce redundant compilation overhead.

[0080] S304, binary dependency propagation step, extracts the dependencies between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived binary code that needs to be reused; then, for the function whose main body is deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused.

[0081] In this embodiment, the old binary file refers to a binary file generated using old source code. The partial data segment refers to the partial data segment that the reused binary dependency graph depends on.

[0082] Specifically, we first discuss the derived binary codes that a function may depend on, and then design a binary dependency graph (BDG) to accurately encode these dependencies. In addition, we design a binary dependency propagation method to identify the derived binary codes of the pruned functions.

[0083] (1) Derived binary code data

[0084] Directly reusing the binary code of a function whose semantics remain unchanged is not enough to ensure the correctness of compilation, because the unchanged source code may still generate additional binary data. Deleting a function may result in its derived code or data not being generated. The binary code data that may be derived from a function is divided into three categories.

[0085] 1) Derived functions: Deleting a function body may prevent the instantiation of templates it uses, or cause the compiler to remove WEAK / GLOBAL functions. Figure 4 For example, deleting the function body of test2 will result in temp <int>Unable to instantiate.

[0086] 2) Derived data: Removing the body of a function will result in no read-only data being generated that it uses. Figure 4 For example, deleting the function body of test2 will cause the read-only data "exp" to fail to be generated.

[0087] 3) Derived sections: In binary code, the semantics of a function consists of sections with different functions. For example, the function's code is stored in the text section, its dependencies are stored in the relocation section, and any exceptions thrown by the function are stored in the exception section. When the body of the function is removed, these derived sections cannot be generated. Figure 4 For example, trimming the function body of test2 will result in the .text.Z5test2v code section.

[0088] The .rela.text.Z5test2v relocation section and the corresponding entry in the exception section .eh_frame cannot be generated.

[0089] (2) Deriving a binary dependency graph

[0090] A Binary Dependency Graph (BDG) is designed to identify derived binary codes that need to be reused. In BDG, nodes represent binary symbols and sections, and edges describe the dependencies between binary symbols and segments, including symbol relocation dependencies and section dependencies.

[0091] 1) Symbol relocation dependency: In a binary file, the relocation table is an ELF section containing relocation entries that help the linker or loader adjust the memory addresses in the binary code to ensure that the program's symbols are correctly referenced at runtime. In other words, the relocation table describes the explicit dependencies between binary symbols. By utilizing symbol relocation dependencies, functions and data derived from a given function can be identified. For example, in Figure 6 In the example, function test2 derives the template function temp instantiated with integer and read-only data "exp". The relocation table .rela.text.Z5test2 will contain corresponding entries pointing to Z4testIiET_S0 and .L.str.

[0092] 2) Binary section dependencies: To identify sections derived from symbols, dependencies are established between symbols and various functional sections, including code sections, data sections, relocation sections, exception sections, and section groups. For example, in Figure 6 In the example, the symbol Z5test2v depends on the code section .text.Z5test2v, the relocation section .rela.text.Z5test2v, and the exception section .eh_frame; the symbol .L.str depends on the data section .rodata.str1.1; and the symbol ZN5Base14implEv depends on the code section .text.ZN5Base14implEv and the section group .group.

[0093] (3) Binary Dependency Propagation

[0094] Similar to the compilation dependency propagation method, the binary dependency propagation method also uses breadth-first search. Starting from the semantically unchanged functions, this method recursively traverses the symbol relocation dependencies and section dependencies on the BDG. If the traversed symbols or sections do not exist in the streamlined binary code, they will be reused.

[0095] exist Figure 6 In the example, starting from the functions ZN5Base14implEv and Z5test2v whose semantics remain unchanged, we can determine that the symbols that need to be reused include ZN5Base14implEv, Z5test2v, .L.str, and Z4tempIiET_S0; in addition, the sections that need to be reused include .group, .text.ZN5Base14implEv, .text.Z5test2v, .rela.text.Z5test2v, .rodata.str1.1,

[0096] .text.Z4tempIiET_S0, and the corresponding frame description entry in .eh_frame. Specifically, since the symbol Z5test1v and section .text.Z5test1v already exist in the partial binary, there is no need to reuse them.

[0097] S305, a binary multiplexing step, merging symbols and some data segments that need to be reused in the old binary file into a partial binary file to form a complete binary file.

[0098] The goal of binary reuse is to ensure the correctness of the binary code generated by incremental compilation. To this end, the deleted functions and their dependencies are reused from the old binary file into the generation of the partial binary file. Binary reuse is divided into two types: real-name symbol reuse and anonymous symbol reuse.

[0099] (1) Real-name symbol reuse

[0100] The symbolic names of real-name symbols serve as unique identifiers, so it is possible to match named symbols between old and new binaries. In addition, sections derived from named symbols also have unique identifiers. If the named symbols or sections that need to be reused do not exist in the partial binary, these symbols and sections will be created. This involves updating the symbol table and section header table, and adding the symbol names to the string table. If these symbols and sections already exist in the partial binary, the corresponding symbol and section indexes will be adjusted to point to the existing symbols and sections. It is important to note that even if the body of a function is pruned, the compiler will still generate an undefined symbol for it. These symbols will be removed before reuse.

[0101] by Figure 7 For example, for the real-name symbol Z5test2v that does not exist in the partial binary, a new entry will be created for it in the symbol table .symtab, and the symbol name "Z5test2v" will be updated in the string table .strtab. In addition, the .text.Z5test2v, .rela.text.Z5test2v sections, and entries corresponding to the .eh frame will be created in the partial binary, and the section header table will be updated. Specifically, the .rela.text.Z5test2v section depends on the real-name symbol Z5test1v, which exists in the partial binary. The relevant index in .rela.text.Z5test2v will be adjusted to point to the existing symbol Z5test1v.

[0102] (2) Anonymous Symbol Reuse

[0103] For read-only data, the compiler generates a randomly named read-only data section as well as an anonymous symbol. In the binary code, anonymous symbols can be identified by section offset or by using a random symbol name. Since anonymous symbols are read-only and will not be modified at runtime, even if two anonymous symbols have different virtual addresses, they are semantically equivalent as long as their binary data is the same. Therefore, for anonymous symbols that need to be reused, a new anonymous symbol and a new read-only data section are created directly in the partial binary file. In order to minimize space waste, the binary data stored by the anonymous symbol will be reused in the new read-only data section when needed. It should be noted that since the names of anonymous symbols and read-only data sections may be random, their names need to be modified to avoid symbol conflicts in partial binaries.

[0104] by Figure 7 For example, for the anonymous symbol .L.str and its related read-only data section .rodata.str1.1, a new anonymous symbol .L.str@1 and a new read-only data section .rodata.str1.1@1 will be generated in the partial binary file. Since .rodata.str1.1 and .rodata.str1.1@1 store the same read-only data "exp", the anonymous symbols .L.str and .L.str@1 are semantically equivalent.

[0105] In summary, the above S304 to S305 of this embodiment solves the problem of correctly reusing the binary code of the function, and can accurately identify the function that has changed in semantics.

[0106] The Binary Dependency Graph (BDG) designed in the above steps is used to describe the dependencies between symbols in the old binary code. By traversing the BDG, the derived binary code of the deleted function can be accurately identified. Subsequently, for each deleted function, its binary code and its derived binary code can be extracted from the old binary file. Finally, the extracted old binary code is merged into the newly compiled binary code.

[0107] See also Figure 2 As shown, according to BDG, the template instantiation function test can be determined <int>is derived from function foo. Therefore, when reusing the binary code of foo, test is reused. <int>The binary code prevents the lack of test <int>This can cause the new binary to crash at runtime.

[0108] The function-level incremental compilation method of the above software project can effectively avoid Figure 1 Specifically, when the method of the present invention is applied to Figure 1 When compiling the example shown, the observed time cost is about 1 second. This time cost is almost the same as the time cost of compiling only the test0 function.

[0109] In this embodiment, the function-level incremental compilation method of the software project of the present invention is implemented by C / C++, namely IClang. IClang uses semantic equivalence detection and binary on-demand reuse, as follows.

[0110] (1) Semantic equivalence detection

[0111] IClang is agnostic to existing file-level incremental compilation methods. When no semantic changes are detected in a source file, IClang directly reuses previously generated binaries. Specifically, if a source file does not use the changed code, IClang considers the semantics of the source file to be unchanged. Moreover, this analysis can be easily performed on CDG without introducing additional analysis overhead.

[0112] (2) Binary on-demand reuse

[0113] Reuse symbols and sections as needed to minimize space redundancy in the new binary. C / C++ compilers usually compile all functions into a single text section and all data into a single data section. Reusing these sections in their entirety can result in significant space waste. Therefore, use the -ffunction-sections and -fdata-sections compilation options to allocate a section for each symbol. For sections that store read-only data in a specific format, such as read-only string tables and read-only floating-point tables, reuse these data as needed according to the corresponding data format. For example, split the read-only string table at the '\0' character to avoid redundant reuse of unrelated strings. In addition, specific reuse strategies are customized for sections with specific structural compositions. For example, the exception table consists of frame description entries (FDEs) from different functions. When reusing a function, only its corresponding FDE is reused into the exception table of the new binary instead of reusing the entire exception table.

[0114] The following will be verified by experiments.

[0115] (1) Experimental setup

[0116] Project Collection: As shown in Table 1, five widely used C / C++ projects are collected, ranging in size from 270KLoC to 2,540KLoC. In order to evaluate the incremental compilation efficiency of IClang, 100 commits are collected for each project. It is ensured that each commit contains code changes that cause semantic changes, and these changes are limited to the function body. Each project has been fully tested in previous studies, and a comprehensive test suite is provided to verify the correctness of IClang. In addition, for each project, 10 fix commits with corresponding verification test cases are collected to verify whether IClang can correctly update the fix semantics in the binary code.

[0117] Table 1 Project data

[0118]

[0119] Benchmark test: The performance improvement of IClang of the present invention is compared with the mainstream C / C++ compiler Clang and the file-level incremental compilation method ABC (Ying Zhang, Yanyan Jiang, Chang Xu, Xiaoxing Ma, and Ping Yu. ABC: Accelerated building of c / c++ projects. In 2015 Asia-Pacific Software Engineering Conference (APSEC), pages 182-189, 2015. https: / / doi.org / 10.1109 / APSEC.2015.27.).

[0120] Experimental environment: Each project is compiled in single-threaded and debug mode. Currently, incremental updates to debug tables are not supported. To ensure a fair comparison, debug tables are disabled in all methods. The experiments are conducted on a server equipped with a 104-core Intel(R) Xeon(R) Gold 6230R CPU@2.10GHz and 500GB memory. The server runs Ubuntu 18.04 system with Linux kernel version 5.4.0-135-generic.

[0121] (2) Experimental results

[0122] Table 2 shows the compilation speedup achieved by IClang and ABC compared to Clang.

[0123] Table 2 Performance comparison of the method of the present invention and other methods

[0124]

[0125] The data shows that IClang exhibits superior efficiency, with an average speedup of 2.34 times and 1.58 times (2.34 / 1.48) relative to Clang and ABC, respectively. IClang performs particularly well in the incremental compilation of SQLite, with an average speedup of 4.05 times. This significant improvement is attributed to SQLite's unique compilation process, which integrates all source code into a single file. Therefore, each submission requires the entire project to be recompiled, and IClang's function-level incremental compilation is effectively optimized in this case.

[0126] Figure 8 The speedup distribution of IClang and ABC compared to Clang is shown. IClang achieves higher speedup than ABC in almost all the submissions evaluated (497 out of 500 submissions).

[0127] See also Fig. 9 As shown, this embodiment also discloses a function-level incremental compilation system for a software project, including:

[0128] The symbol change analysis module 901 is used to identify functions with text changes and extract symbols used in the changed code through lexical analysis;

[0129] A compilation dependency propagation module 902 is used to perform compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and symbols used in the changed code to identify the functions with semantic changes;

[0130] A selective compilation module 903 is used to delete the body of the function that has not changed semantically, generate only part of the source code, and then compile to generate part of the binary file;

[0131] The binary dependency propagation module 904 is used to extract the dependency relationship between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived symbols and partial data segments that need to be reused; then, for the function whose main body is deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused;

[0132] The binary multiplexing module 905 is used to merge the symbols and partial data segments that need to be reused in the old binary file into the partial binary file to form a new and complete binary file.

[0133] Specific Implementation of a Function-Level Incremental Compilation System for a Software Project The function-level incremental compilation method for the same software project will not be repeated in this embodiment.

[0134] The principle of the present invention and its operation mode are explained through the above-mentioned specific implementation cases. These cases are intended to provide readers with a clear understanding framework to grasp the core ideas and operation points of the present invention. However, it should be clear that these cases are not limitations on the scope of application of the present invention. For professionals in this field, various forms of improvements and innovations based on the core ideas of the present invention should be included in the protection scope of the present invention.< / int> < / int> < / int> < / int> < / float> < / float> < / int> < / int>

Claims

1. A function-level incremental compilation method for a software project, characterized in that: include: The symbol change analysis step identifies functions with text changes and extracts the symbols used in the changed code through lexical analysis; The compilation dependency propagation step performs compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and the symbols used in the changed code to identify the functions that have changed semantically; A selective compilation step that prunes the bodies of semantically unchanged functions, generating only partial source code that is then compiled to produce partial binaries; The binary dependency propagation step extracts the dependencies between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived symbols and partial data segments that need to be reused. Then, for the functions whose bodies have been deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused. The binary reuse step merges the symbols and partial data segments that need to be reused in the old binary file into the partial binary file to form a new and complete binary file.

2. The function-level incremental compilation method for a software project according to claim 1, characterized in that: The symbol change analysis step specifically includes: Preprocess the source code and identify the text differences between the new and old preprocessed codes; if the text of the function body changes, it is determined that the semantics of the function has changed; Lexical analysis is performed to extract identifiers in the textual change code, and it is assumed that symbols with the same names as these identifiers will be used by the change code.

3. The function-level incremental compilation method for a software project according to claim 1, characterized in that: The compilation dependency graph CDG includes nodes and edges, wherein nodes represent symbols in source code; edges are used to describe dependencies between symbols, including resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies.

4. The function-level incremental compilation method for a software project according to claim 3, characterized in that: The compilation dependency propagation specifically includes: Starting from the functions with changed text and the symbols used by the changed code, recursively traverse the resolved symbol dependencies, unresolved symbol dependencies, and implicit symbol dependencies on the CDG; After the propagation is completed, if the function meets one of the following two conditions, the semantics of the function is considered unchanged; the two conditions include: This function has not been accessed during the propagation process; The function was accessed during propagation, but the function's text was not changed, the function was not a template function, and the binary code of the function existed in the previous binary.

5. The function-level incremental compilation method for a software project according to claim 1, characterized in that: The selective compilation step also includes: during the compilation process, using semantic analysis to eliminate lexical errors and dependencies.

6. The function-level incremental compilation method for a software project according to claim 1, characterized in that: The binary dependency graph BDG includes nodes and edges, wherein the nodes represent binary symbols and sections; and the edges are used to describe the dependency relationship between binary symbols and segments, including symbol relocation dependency and binary section dependency.

7. The function-level incremental compilation method for a software project according to claim 6, characterized in that: The binary dependency propagation step adopts breadth-first search; starting from the function with unchanged semantics, symbol relocation dependencies and section dependencies are recursively traversed on BDG; if the traversed symbols or sections do not exist in some binary files, they will be reused.

8. The function-level incremental compilation method for a software project according to claim 1, characterized in that: In the binary multiplexing step, the binary multiplexing includes real-name symbol multiplexing and anonymous symbol multiplexing; The symbolic name of the real-name symbol serves as a unique identifier, enabling matching of named symbols between old binaries and partial binaries; In addition, sections derived from named symbols also have unique identifiers; if the named symbols or sections that need to be reused do not exist in the partial binary file, these symbols and sections are created, the symbol table and section header table are updated, and the symbol names are added to the string table; if these symbols and sections already exist in the partial binary file, the corresponding symbol and section indexes are adjusted to point to the existing symbols and sections; For read-only data, the compiler generates a randomly named read-only data section and an anonymous symbol. In the binary, anonymous symbols are identified by section offsets or by using random symbol names. For anonymous symbols that need to be reused, a new anonymous symbol and a new read-only data section are directly created in some binary files. The binary data stored by the anonymous symbol will be reused in the new read-only data section when needed.

9. A function-level incremental compilation system for a software project, characterized in that: include: The symbol change analysis module is used to identify functions with text changes and extract the symbols used in the changed code through lexical analysis; A compilation dependency propagation module is used to perform compilation dependency propagation on the compilation dependency graph CDG of the old source code based on the changed functions and symbols used in the changed code to identify the functions that have changed semantically; A selective compilation module that prunes the bodies of semantically unchanged functions, generating only partial source code that is then compiled to produce partial binaries; The binary dependency propagation module is used to extract the dependencies between symbols from the old binary file to generate a binary dependency graph BDG to identify the derived symbols and partial data segments that need to be reused; then, for the function whose main body is deleted, binary dependency propagation is performed on the binary dependency graph BDG to determine the symbols and partial data segments that need to be reused; The binary reuse module is used to merge the symbols and partial data segments that need to be reused in the old binary file into the partial binary file to form a new and complete binary file.

Images