Collection makefile generator

Abstract

Collection makefile generators generate comprehensive makefiles for processing collections of computer files. In operation, the present collection makefile generator dynamically discovers collection content files, classifies them according to content type and required processing actions, and then generates a makefile for performing those actions. Importantly, all build order dependencies are properly maintained among multiple collection products and among individual collection content files. Automated collection makefile generators drastically improve the productivity of human workers by effectively reducing makefile creation and maintenance costs to zero. Collection makefile generators thus enable humans to easily generate complex makefiles in an automated, scalable way that was not previously possible.

Description

CROSS REFERENCE TO OTHER APPLICATIONS [0001] This application is a continuation of U.S. patent application Ser. No. 09 / 885,077 (Attorney Docket No. CFSTP003) entitled COLLECTION MAKEFILE GENERATOR, filed Jun. 21, 2001, which is incorporated herein by reference for all purposes. CROSS REFERENCE TO RELATED APPLICATION [0002] This application is related to U.S. patent application Ser. No. 09 / 885,076 (Attorney Docket No. CFSTP001) entitled COLLECTION CONTENT CLASSIFIER, filed Jun. 21, 2001, now U.S. Pat. No. 6,934,694, which is incorporated herein by reference in its entirety, for all purposes. This application is also related to U.S. patent application Ser. No. 09 / 885,078, (Attorney Docket No. CFSTP002) entitled COLLECTION INFORMATION MANAGER, filed Jun. 21, 2001, which is incorporated herein by reference in its entirety, for all purposes.FIELD OF THE INVENTION [0003] This invention relates to automated software systems for processing collections of computer files in arbitrary ways, th...

AI Technical Summary

Problems solved by technology

Unfortunately, replacing arbitrary manual processes performed on arbitrary computer files with automated systems is a difficult thing to do.

As a consequence, the general software productivity problem has not been solved yet, despite large industry investments of time and money over several decades.

The main problem to be solved by the original make program was that humans could not reliably figure out which source files needed to be recompiled-after each program source code modification was made.

Specifically, humans could not easily keep track of the various interdependencies that typically existed among multiple source files.

Missing dependency relationships frequently lead to failed compilations, incorrect results, wasted time, and overall lower software development productivity.

This was very costly in terms of computer resources and wasted human programming time.

Unfortunately, make programs give rise to another significant productivity problem, which is the ubiquitous problem of manually creating and maintaining makefiles.

Manually creating and maintaining makefiles is time consuming, costly, and error prone for several reasons.

First, a significant amount of human time is required for programmers to first learn about make programs and makefiles.

The knowledge burden imposed on individual programmers is consequential, especially if advanced or complex features of make programs must be understood.

Second, creating makefiles typically requires that programmers manually list all source files, dependencies, processing commands, processing variations, and makefile targets that are involved in make operations.

However, the requirements rapidly become very demanding, time consuming, and complex as the number of source files, dependencies, performed command sequences, process variations, and makefile targets increase in number.

Third, software maintenance costs caused by ongoing development activities are significant, especially for makefiles that are used to manage medium or large software systems.

Fourth, evolutionary changes in computer environments often cause precision makefiles to “break” in some way.

Since makefiles describe precise, complex processes, even small changes in computing environments can generate disproportionately large makefile maintenance costs.

Fifth, human programming mistakes or modifications that “break” makefiles may trigger many downstream costs, ranging from wasted program test runs to increased makefile debugging costs.

For example, it is easy for humans to miss a dependency, omit a source file, or make a mistake when working on complex makefiles for large software systems.

These increased downstream costs can be significant, ranging from trival losses of a few minutes here and there on small projects to consequential losses of several days or weeks on large, more complex projects where feedback cycle times are longer.

As can be seen from the above, manual makefile techniques clearly lower human productivity.

The makefile generator problem is very difficult, primarily because of the large amounts of variance within every dimension of the makefile problem.

Further practical complications include using arbitrary computing platforms, arbitrary software toolsets, and arbitrary administrative policies.

A final complication is that many of the factors listed above are coupled, so that decisions in one dimension affect decisions in other dimensions.

For example, choosing to use a particular software tool may affect the design of the overall processing sequence.

Since even human programmers have practical difficulties working with such makefiles, constructing automated makefile generators to produce makefiles of similar complexity is obviously difficult.

It is the problem how to model, manage, and provide collection instance information, collection content file information, and collection data type information for eventual use by application programs that process collections.

It is the problem of how to determine collection content members, content types, content processing actions, and content processing interdependencies.

It is the problem of how to automatically calculate and generate a precision makefile for managing the efficient application of complex computer command sequences to various collections of computer files.

It is the problem of how to ensure that multiple products within one collection are processed in correct dependency order to ensure proper software build results.

It is the problem of how to ensure that particular files within one product within one collection are processed in correct dependency order to ensure proper software build results.

It is the problem of ensuring that there is a one-to-one match between (a) the include files that are found using makefile generator search rules and that are subsequently listed as dependencies within the makefile, and (b) the include files that are found using compiler search rules at compiler runtime.

If a mismatch occurs, an incorrect build sequence or a wasteful build sequence may occur.

It is the problem of ensuring that there is a one-to-one match between (a) the library files that are found by makefile generator library search rules and that are subsequently listed as dependencies within a makefile, and (b) the library files that are found by linker search rules at linker runtime.

If a mismatch occurs, an incorrect build sequence or a wasteful build sequence may occur.

It is the problem of managing name conflicts within makefiles that build multiple products from the same set of source files, where the build command sequences differ among products.

It is the problem of how to optimally use available parallel processing power to perform makefile operations in a minimum amount of time.

Some interesting aspects of the Parallel Makefile Target Problem are these: there is an inherent limit to the amount of parallelism that can be achieved within each collection of files to be processed; there is a physical limit to the amount of parallel processing power available in each computational environment; and there is a policy limit to the amount of parallelism that can be used by makefiles in each administrative environment.

It is the problem of how to optimally share makefile generator template files among various computing platforms to maximize software reuse and minimize software maintenance costs.

If they cannot, a general solution to the makefile generation problem cannot be achieved.

As the foregoing discussion suggests, makefile generation is a complex problem.

No competent general solution to the overall makefile generation problem is visible in the prior art today, even though the first make program was created in the 1970s, well over two decades ago.

Prior art approaches lack support for collections.

This is the largest limitation of all because it prevents the use of high-level collection abstractions that can significantly improve productivity.

Prior art approaches lack automated support for dynamically determining lists of collection content files to be processed by makefiles, thereby requiring humans to manually construct content file lists, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for multiple software products that are to be produced from the same collection of files, thereby requiring humans to manually create makefile code for multiple products, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for determining relative build order among multiple software products that are to be produced from the same collection of files, thereby requiring humans to manually declare relative build orders for multiple products, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for resolving name conflicts within makefiles that produce multiple software products from the same set of source files, thereby requiring humans to manually repair name conflicts, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for dynamically locating include files to participate in dependency relationships within the makefile, thereby requiring humans to manually declare such dependencies, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for dynamically locating library files to participate in dependency relationships within the makefile, thereby requiring humans to manually declare such dependencies, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for dynamically determining dependencies in arbitrary programming languages, thereby requiring humans to manually declare such dependencies, and thereby increasing makefile creation and maintenance costs.

Prior art approaches lack automated support for generating makefiles that support parallel execution behavior, thereby preventing the general use of parallel computing capacity to reduce makefile execution times.

Prior art approaches lack well-structured support for sharing makefile templates among across multiple computing platforms, thereby requiring multiple copies of makefile template information, and thereby increasing software maintenance costs.

Prior art approaches lack well-structured support for modelling large ranges of process variance and makefile customizations found within industrial software environments, thereby preventing the widespread use of fully automated makefile generators within industrial environments.

Notably, general prior art mechanisms do not provide fully automated support for collections, dynamic determination of content files, multiple products, extensive makefile variance, or parallel execution support.

Neither does it have a mechanism for sharing content classification information, so multiple automake files cannot easily share user-provided policy information.

Finally, it uses an input file that must be manually constructed, and so its classification operations are not fully automated.

Imake has no support for dynamic content discovery; no automated support for multiple products, or for parallel targets.

Finally, it uses an input file that must be manually constructed, and so its classification operations are not fully automated.

However, only the current directory is used to find source files; no other directories are supported.

The makefile generator approach described by Thompson in U.S. Pat. No. 5,872,977 has no support for dynamic content discovery; no automated support for multiple products, or for parallel targets.

Finally, it uses a platform-independent input build file that must be manually constructed, and so its classification operations are not fully automated.

However, IDEs typically have no support for dynamic content discovery; no fully automated support for multiple products (human interaction is typically required), no support for parallel targets; and no support for collections in general.

Images