A GNU make-based build dependency error detection method

Abstract

The application discloses a kind of based on GNUMake's construction dependency error detection method, comprising: static dependency analysis, actual dependency analysis and dependency error detection.The static dependency analysis part includes GNUMake internal data analysis and establishes static dependency graph;The actual dependency analysis part includes construction process tracking and establishes actual dependency graph;The dependency error detection part includes construction target unification and dependency analysis.The method disclosed by the application efficiently and simply helps developer to complete the construction script analysis based on GNUMake, detects the dependency error in construction script.

Description

Technical Field

[0001] This invention relates to the field of software engineering, and more specifically to a method for detecting build dependency errors based on GNU Make. Background Technology

[0002] In modern software development, build systems (such as GNU Make) are typically needed to automatically transform human-readable source code into executable software or deliverables. There are many types of build systems, suitable for different programming languages ​​and platforms. Software building is the process of producing deliverables or executable code from source code and related libraries. Build systems coordinate the execution of different compilers and other tools to compile source code, libraries, and other data into executable programs. Therefore, a build process may involve thousands of build commands, which must be executed in a specific order to produce a correct set of deliverables. Build systems describe the process of assembling software by coordinating compilers, preprocessors, and other tools, allowing developers to focus on changes to the software source code.

[0003] Common build systems typically require build scripts to build software correctly and efficiently. Build scripts contain build targets, build dependencies, build commands, and build order. Initially, the build system executes a full build based on the build script to compile all targets; this process is called a full build. Modern software development pursues high efficiency and high quality, so continuous integration and continuous deployment have become mainstream development paradigms, potentially involving multiple build rounds in a short period. However, in some large projects, a single full build can take a considerable amount of time (for example, a full OpenCV build takes approximately 20 minutes). Therefore, software engineers often use incremental and parallel builds to save time in each build round. For example, when local code changes or is updated to the latest files in the version control system, incremental builds can be used to rebuild only targets with at least one changed dependency, significantly reducing the time required to wait for the build to complete. Most build systems, including GNUMake, Bazel, and Ninja, support advanced features such as incremental and parallel builds to accelerate the build process. In incremental builds, historical build results are reused, rebuilding only a subset of targets—the modified files. In parallel builds, the build system first analyzes the build script to determine which targets can be built simultaneously and assigns tasks to different CPU cores to achieve shorter build times.

[0004] For incremental and parallel builds to execute correctly, the build system needs to understand the dependencies between build targets. Build dependencies are described by build scripts, which play a crucial role in the build process. However, for a large-scale software project, the dependencies between target files are complex. While some existing tools like CMake and Autotools can automatically generate build scripts, software engineers sometimes need to manually enumerate all dependencies. Dependency enumeration for large projects is a time-consuming, tedious, and error-prone process. Failure to enumerate the correct dependencies can hinder the execution of incremental and parallel builds. Summary of the Invention

[0005] To address the aforementioned problems and challenges, this invention proposes a build dependency error detection method based on GNU Make, which helps developers quickly detect dependency errors in build scripts.

[0006] A build dependency error detection method based on GNU Make includes: static dependency analysis, actual dependency analysis, and dependency error detection;

[0007] The static dependency analysis section includes GNUMake database parsing and static dependency graph construction;

[0008] The actual dependency analysis part includes building monitoring and constructing the actual dependency graph;

[0009] The dependency error detection part includes dependency graph unification and error detection.

[0010] The specific steps of the static dependency analysis section include:

[0011] (1-1) Start the build by using the debug command Make-p to get internal data.

[0012] (1-2) Read the Files section of the database, which lists custom and suffix rules associated with a specific file. Intermediate file and suffix rules are marked as "not a target," while the rest are marked as "targets." Each file contains comments explaining how Make should process this rule.

[0013] (1-3) Perform text analysis and resolve static dependencies.

[0014] Step (1-1) The static dependency graph comes from the rules defined in the build script, which contains the dependencies between different targets and source code files. The build process requires the build system (such as GNU Make) to read the build script (such as a Makefile), parse the build commands, and then execute the build. When GNU Make executes the build, the build script is parsed and stored in an internal database. GNU Make provides debugging functionality, which can be obtained using the command `make -p`.

[0015] The data in step (1-2) base can be divided into the following sections: Variables, Directories, Implicit Rules, Pattern-specific variable values, Files, and VPATH Search Path. Variables lists each variable with descriptive comments. The next section is Directories, which lists the directories that Make will inspect. For each directory, Make displays implementation details such as device number, inode, and filename pattern matching statistics. Implicit Rules is the third section, containing all built-in and user-defined pattern rules in the Make database. Additionally, for rules defined in files, their comments indicate filenames and line numbers. The section marked Pattern-specific variable values ​​lists pattern-specific variables defined in the Makefile. Pattern-specific variables are variables whose scope is limited to the execution of the associated pattern rule. The Files section lists custom and suffix rules associated with specific files. Intermediate file and suffix rules are marked "not a target," while the rest are marked "targets." Each file contains comments explaining how make processes this rule. Found files are displayed using a standard VPATH search to find their paths. The last part, marked as VPATH Search Path, lists the values ​​of VPATH and all VPATH modes.

[0016] Steps (1-3) obtain static dependencies by parsing the contents of the Files section, creating a node for each build target that contains all the dependencies required by that target. Even if the recipe for a target is empty, an empty node is created using the target name. Finally, a static dependency graph with targets as nodes and targets as edges can be generated.

[0017] The actual dependency analysis section includes:

[0018] (2-1) Monitor the build process and the system calls used.

[0019] (2-2) During the build process, the files read and written by the build process are analyzed to construct the actual dependency graph.

[0020] Step (2-1) In this paper's approach, ptrace is used to obtain the actual dependencies from operations during the file tracing process. In Linux, ptrace provides a way for the parent process to monitor and control other processes. It also allows modification of registers and the kernel image in child processes, enabling breakpoint debugging and system call tracing. This allows this paper to monitor underlying hardware and services (such as the file system). The actual build process is obtained and recorded by monitoring and recording file system calls during the build process.

[0021] Step (2-2) builds a record that stores the filenames read, created, written, and renamed by each process, and establishes associations from each input file to the output file. After a complete and error-free compilation, the actual dependency graph can be generated. Nodes in the graph represent input and output files, and edges represent process calls. Process information will be removed to represent the actual dependencies.

[0022] The dependency error detection component includes:

[0023] (3-1) Dependency graph unification;

[0024] (3-2) Error detection;

[0025] Step (3-1) shows that the static dependency graph reflects the dependencies declared in the build script, while the actual dependency graph reflects the actual dependencies during the build process. For the same build target, incorrect dependencies can be identified by comparing the differences between the two dependency graphs. However, nodes in the actual dependency graph are represented as filenames, while nodes in the static dependency graph are represented as targets, which leads to inconsistencies. Generally, in the actual dependency graph, the build target is represented as the output file. Some nodes in the static dependency graph also contain relative paths, which can be represented as $(DESTDIR) / TARGET. Nodes in the actual dependency graph use absolute paths. The presence of relative paths is detrimental to subsequent node matching and should be removed, and the nodes should be normalized. Based on the node structure of the static dependency graph, nodes in the actual dependency graph can be matched using the trailing characters.

[0026] Step (3-2) primarily detects missing and redundant dependencies in the build script. Redundant dependencies arise from dependencies declared in the build script that are not used in the actual build. Redundant dependencies of the target mean that edges contained in the static dependency graph will not appear in the edges of the actual dependency graph. In our method, we check whether the edges of each target node in the static dependency graph appear on the edges of the corresponding nodes in the actual dependency graph. Static dependencies that do not match the actual dependencies are reported as redundant dependencies. Missing dependencies originate from dependencies required by the build target but not declared in the build script. In our method, we scan whether the dependencies required by the target node in the actual dependency graph are faithfully reflected in the static dependency graph. Dependencies that do not match in the static dependency graph are reported as missing dependencies of the target. Furthermore, since the edges of nodes in the actual dependency graph contain all the dependencies required for compilation, some dependencies do not come from the source project (e.g., GCC libraries). To reduce false positives, we only report missing dependencies from the source project directory.

[0027] Compared to existing technologies, this solution has the following advantages:

[0028] Completeness of static parsing: This solution is based on parsing the build script by the build system, which can achieve a more complete parsing result compared to directly parsing the build script.

[0029] Effectiveness of dependency error detection: This solution can obtain more complete static and actual dependencies, so its effectiveness in detecting dependency errors is higher.

[0030] Efficiency of dependency error detection: Cross-referencing actual dependencies with static dependencies can complete the detection faster. Attached Figure Description

[0031] Figure 1 This is a complete flowchart of a build dependency error detection method based on GNU Make proposed in this invention;

[0032] Figure 2 This is a complete flowchart of the static dependency analysis stage proposed in this invention;

[0033] Figure 3 This is a complete flowchart of the actual dependency analysis proposed in this invention;

[0034] Figure 4 This is a complete flowchart of the error detection-dependent process proposed in this invention. Detailed Implementation

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings, and not all of them.

[0036] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. It should be clear that while all steps are sequential in the figures, this does not mean that the order in which these steps occur strictly as shown in the figures.

[0037] For ease of understanding, the main inventive concepts of the embodiments of the present invention will be briefly described.

[0038] Example 1

[0039] See Figure 1 The present invention proposes a build dependency error detection method based on GNU Make. This method analyzes build dependency errors in GNU Make projects and outlines the necessary steps and precautions. Specifically, it includes the following steps:

[0040] Step S100, as follows Figure 2 As shown, the build script is analyzed to construct a static dependency graph. This includes the following steps:

[0041] Step S101: Start the build process and obtain the GNUMake internal database using Make-p;

[0042] Step S102: Parse the Files section of the database to find the text information line that is the construction target. The first field of the line is the construction target, followed by the dependencies of the target.

[0043] Step S103: Create a node for each building target, with the edges of the node representing dependencies, and construct a static dependency graph.

[0044] The above steps S101-S103 are a detailed explanation of step S100.

[0045] Example 2

[0046] See Figure 3 As mentioned above, after completing the static dependency graph construction, actual dependency analysis needs to be performed, specifically including the following steps:

[0047] Step S200, as follows Figure 3 As shown, the actual dependency analysis includes the following steps:

[0048] Step S201: Use the ptrace command in Linux to monitor the build process, and monitor all system calls of this process and its child processes.

[0049] Step S202 involves analyzing all process calls to the file system, parsing their file operations, including reading, writing, renaming, and all other file operations. By linking processes to their input and output files, process information is hidden; the relationships between files represent the actual dependencies.

[0050] The above steps S201-S202 are a detailed explanation of step S200.

[0051] See Figure 4 The dependency error detection proposed in this embodiment of the invention specifically includes the following steps:

[0052] Step S300, as follows Figure 4 As shown, error detection relies on specific steps, including:

[0053] Step S301: Unify the static dependency graph and the actual dependency graph by matching the relative paths in the static dependency graph with the absolute paths in the actual dependency graph.

[0054] Step S302: Check whether the edge of the target node in each static dependency graph appears on the edge of the node in the actual dependency graph corresponding to the target. Static dependencies that do not match the actual dependencies are reported as redundant dependencies.

[0055] Step S303: Scan whether the dependencies required by the target node in the actual dependency graph are faithfully reflected in the static dependency graph. Dependencies that do not match in the static dependency graph will be reported as missing dependencies of the target.

[0056] The above steps S301-S203 are a detailed explanation of step S300.

[0057] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or alternatives made based on the above technical solutions shall all fall within the scope of protection of the present invention.

Claims

1. A build dependency error detection method based on GNU Make, characterized in that, include: The method includes the following steps: Step 1: Static dependency analysis, analyzing the build target by analyzing its build scripts and static dependency graph through GNUMake's internal database; Step 2: Actual dependency analysis, which tracks and monitors the actual build of the target and constructs an actual dependency graph using input and output files; Step 3: Dependency error detection. Unify the actual dependency graph and the static dependency graph, and then perform error detection. Step 1, static dependency analysis, is as follows: (1-1) Start the build by using the debug command Make -p to obtain internal data; (1-2) Read the Files section of the database. This section lists the custom and suffix rules associated with a specific file. Intermediate files and suffix rules will be marked as "not a target", while the rest will be marked as "target". Each file contains comments explaining how Make processes this rule. (1-3) Perform text analysis and resolve static dependencies; Step (1-1) The static dependency graph comes from the rules defined in the build script, which contains the dependencies between different targets and source code files. The build action requires the build system to read the build script, parse the build commands, and then execute the build. When GNU Make executes the build, the build script is parsed and stored in an internal database. GNU Make provides debugging functionality and can obtain internal data through the command make -p. The data in step (1-2) base can be divided into the following parts: Variables, Directories, Implicit Rules, Pattern-specific variable values, textit{Files}, and VPATHSearch Path. Variables lists each variable with descriptive comments. The next part is Directories, which lists the directories that Make will check. For each directory, Make displays implementation details, including device numbers, inodes, and filename pattern matching statistics. Implicit Rules is the third part, containing all built-in and user-defined pattern rules in the Make database. Additionally, for rules defined in files, their comments indicate filenames and line numbers. The section marked Pattern-specific variable values ​​lists pattern-specific variables defined in the Makefile. Pattern-specific variables are variables whose scope is limited to the execution of the relevant pattern rule. The Files section lists custom and suffix rules associated with specific files. Intermediate files and suffix rules are marked "not a target," while the rest are marked "targets." Each file contains comments explaining how make processes this rule. Found files are displayed using a standard VPATH search to find their paths, marked as VPATH Search. The last part of Path lists the values ​​of VPATH and all VPATH modes; Steps (1-3) obtain static dependencies by parsing the contents of the Files section, create a node for each build target containing all the dependencies required by that target, and create an empty node using the target name even if the recipe for the target is empty. Finally, a static dependency graph with targets as nodes and targets as edges can be generated.

2. The method for detecting build dependency errors based on GNU Make according to claim 1, characterized in that, The static dependency analysis includes GNUMake database parsing and static dependency graph construction.

3. The method for detecting build dependency errors based on GNU Make according to claim 1, characterized in that, The actual dependency analysis includes building monitoring and constructing the actual dependency graph.

4. The method for detecting build dependency errors based on GNU Make according to claim 1, characterized in that, The dependency error detection includes dependency graph unification and error detection.

5. A build dependency error detection method based on GNU Make according to claim 2, characterized in that, GNUMake internal database analysis involves analyzing the parsed build scripts output during the project build process using GNUMake's debugging features to extract static dependency graphs.

6. The build dependency error detection method based on GNU Make according to claim 3, characterized in that, Actual dependency analysis involves analyzing the build process through system calls to construct an actual dependency graph.

7. The build dependency error detection method based on GNU Make according to claim 4, characterized in that, Dependency error detection unifies the build targets in the actual dependency graph and the static dependency graph, and detects dependency errors in the targets.

8. A build dependency error detection method based on GNU Make according to claim 1, characterized in that, Step 2: Actual dependency analysis, as follows: (2-1) Monitor the build process and the system calls used; (2-2) During the build process, the files read and written by the build process are analyzed to construct the actual dependency graph; Step (2-1) In the method described in this paper, ptrace is used to obtain the actual dependencies from the operations in the file tracing process. In Linux, ptrace provides a way for the parent process to monitor and control other processes, and can also change the registers and kernel image in the child process to achieve breakpoint debugging and system call tracing. Step (2-2) builds a record that saves the filenames read, created, written, and renamed by each process, and establishes the association from each input file to the output file. After a complete and error-free compilation, the actual dependency graph can be generated. The nodes in the graph represent input and output files, and the edges represent process calls. Process information will be removed to represent the actual dependency relationships.

9. A build dependency error detection method based on GNU Make according to claim 1, characterized in that, Step 3: Rely on error detection, as follows: (3-1) Dependency graph unification; (3-2) Error detection; Step (3-1) The static dependency graph reflects the dependencies declared in the build script, while the actual dependency graph reflects the actual dependencies during the build process. The nodes in the actual dependency graph use absolute paths. The existence of relative paths is not conducive to subsequent node matching, so they should be removed. The nodes are normalized. As can be seen from the node structure of the static dependency graph, the nodes in the actual dependency graph can be matched by the last character. Step (3-2) mainly detects missing and redundant dependencies in the build script. Redundant dependencies come from dependencies declared in the build script that are not used in the actual build. Redundant dependencies of the target mean that the edges contained in the static dependency graph will not appear in the edges of the actual dependency graph. In the method of this paper, we check whether the edges of the target node in each static dependency graph appear on the edges of the nodes in the actual dependency graph corresponding to the target. Static dependencies that do not match the actual dependencies are reported as redundant dependencies. Missing dependencies come from dependencies required by the build target but not declared in the build script. In the method of this paper, we scan whether the dependencies required by the target node in the actual dependency graph are faithfully reflected in the static dependency graph. Dependencies that do not match in the static dependency graph will be reported as missing dependencies of the target.

Images